AI paper route: Attention, FlashAttention, LoRA, DPO, RAG

Why it matters

AI research and research-engineering interviews often include a paper discussion: one you choose, one from a reading list, or the paper behind a technique on your CV. Interviewers look for whether you understood the mechanism, can tell what was proven from what was measured, and can say what the evidence does not show. Skimming every popular paper prepares you for none of that. This route picks six papers that anchor today's LLM stack and says how deeply to read each for your target role.

Core ideas

Foundations first. The CS229 main notes cover backpropagation in the deep-learning chapter and, in newer chapters, foundation models (including a LoRA section), retrieval-augmented generation, large language models and reinforcement learning; use them to fill gaps before each paper. CS336 rebuilds a language model from scratch: tokenizer, transformer and optimizer (Assignment 1), profiling and a FlashAttention-2 kernel (Assignment 2), scaling (3), data (4), alignment and RL (5). If you can do the matching CS336 assignment, the paper will read easily.

What to extract from each paper.

• Attention Is All You Need (2017): scaled dot-product and multi-head attention, positional encodings, the per-layer complexity comparison with recurrence and convolution, the training recipe (warmup schedule, label smoothing) and the model-variation table, which acts as the ablation study.
• FlashAttention (2022): exact attention computed in tiles with an online softmax and recomputation in the backward pass, an IO analysis that counts reads and writes to GPU high-bandwidth memory, and wall-clock benchmarks. Read it as an argument about memory traffic, not FLOPs.
• PagedAttention / vLLM (2023): the KV cache stored in fixed-size blocks with a block table, as in virtual-memory paging, which nearly removes fragmentation, enables sharing across sampled sequences and supports preemption by swapping or recomputation. Evidence is serving throughput at comparable latency on synthesized request traces.
• LoRA (2021): frozen pretrained weights plus a trainable low-rank update BA, initialized so the model starts unchanged and mergeable for inference. The analysis of which matrices to adapt and how small the rank can be matters as much as the main tables.
• DPO (2023): starting from RLHF's KL-regularized objective, express the reward through the optimal policy and a reference policy, then fit preference pairs with a logistic loss on log-probability ratios. Study the derivation and the roles of the reference model and β.
• RAG (2020): a dense retriever and a sequence-to-sequence generator with retrieved passages treated as a latent variable, marginalized per sequence or per token, evaluated on open-domain QA, generation and fact verification, plus index hot-swapping.

Which roles need which papers.

• Research-oriented ML and AI research: Attention, LoRA and DPO at working or deep level; RAG at working level; FlashAttention if you work on efficiency.
• AI infrastructure: FlashAttention and PagedAttention deep, Attention working, LoRA for multi-adapter serving, DPO and RAG at awareness.
• ML engineering: Attention and LoRA working, RAG working if your team builds retrieval, the rest at awareness.
• Applied AI engineering: RAG at working depth plus current retrieval practice beyond it; Attention and LoRA at awareness; FlashAttention, PagedAttention and DPO optional. These are reading priorities, not predictions about any particular interview.

Three passes. First pass (20–30 minutes): abstract, figures, conclusion; write the claim in one sentence. Second pass: method and evaluation, filling the structured notes. Third pass, for deep reads only: re-derive or reimplement the mechanism and reproduce one small result.

Common misconceptions

• "Every AI role needs all six papers in full." Depth should follow the role; applied-AI interviews probe RAG engineering far more than FlashAttention's IO analysis.
• "FlashAttention approximates attention to save memory." It is exact; savings come from never materializing the attention matrix in HBM and recomputing it in the backward pass.
• "DPO optimizes a different objective because it drops the reward model." It targets the same KL-regularized objective under the Bradley-Terry model; the reward is implicit in the policy and reference.
• "LoRA always matches full fine-tuning." The paper reports specific models and tasks; the gap depends on task, rank, adapted matrices and budget.
• "Today's RAG systems are what the RAG paper describes." It trains retriever and generator jointly on fixed tasks; most current systems use frozen retrievers and prompted LLMs.
• "PagedAttention makes the attention kernel faster." Its gain is memory efficiency that allows larger batches; the paged kernel itself adds some overhead, which the paper measures.

How to approach it

Read your role's deep-read papers first, alternating each with the matching CS229 or CS336 material. Use the paper guides' note prompts, and for every paper practise a two-minute and a ten-minute explanation. Interviewers often follow up with "what would you change?" or "what does this result not show?", so end each paper with one experiment you would run next.