Fine-tuning, LoRA, instruction tuning and preference optimization

Why it matters

Fine-tuning is expensive to do well and easy to do badly, so interviewers probe judgment as much as mechanics: when it is justified, what it changes in the model, and how you would know it worked. Research roles add the derivation of preference-optimization objectives.

Core ideas

When not to fine-tune. Fine-tuning is effective at changing behavior, style and format; it is a poor way to add fresh or frequently changing facts, and it creates a model you must evaluate, version and serve. Try, in order: a clear prompt with examples, retrieval for knowledge, tools for computation and actions, and an evaluation set that tells you whether any of these work. Fine-tune when you need a consistent format or style at scale, a skill the base model lacks in your domain, lower latency or cost through a smaller specialized model, or behavior that prompting cannot elicit reliably. Without an evaluation set, you cannot tell whether a fine-tune helped.

Full versus parameter-efficient fine-tuning. Full fine-tuning updates every weight and needs memory for weights, gradients and two Adam moment tensors, roughly four times the weight memory at one precision before activations. Parameter-efficient methods freeze the base model and train a small set of new parameters.

LoRA. For a frozen weight W0 of shape (d, k), LoRA learns delta W = B A with B of shape (d, r) and A of shape (r, k), where the rank r is much smaller than d and k, scaled by alpha / r. B starts at zero, so training begins exactly at the base model. Only A and B receive gradients and optimizer state, so memory and checkpoint size drop sharply and many task adapters can share one base model. After training, W0 + (alpha / r) B A can be merged into the base weights for zero inference overhead, or adapters can be kept separate and swapped per request. Attention projections are the usual targets, often with the MLP projections too. QLoRA keeps the frozen base in 4-bit precision while training the adapters in higher precision.

Supervised fine-tuning (SFT). Train on prompt-response pairs with the same next-token loss as pretraining, usually computing the loss only on response tokens and using exactly the chat template the model will see in production. Quality and consistency matter more than volume; a small clean set often beats a large noisy one. Watch for memorized phrasing, forgetting of general skills, and format drift when training and serving templates differ.

RLHF. The classic pipeline, described in the InstructGPT paper: (1) SFT on demonstrations; (2) collect human comparisons between model outputs and train a reward model with a pairwise loss such as -log sigmoid(r(x, y_w) - r(x, y_l)); (3) optimize the policy with PPO to maximize reward minus a KL penalty to the SFT reference model. The KL term keeps the policy where the reward model is trustworthy; without it, the policy finds outputs that exploit reward-model errors (reward hacking). PPO needs on-policy sampling, a value model and careful tuning, which makes it expensive and brittle.

DPO. DPO starts from the same KL-regularized objective, writes the optimal policy in closed form, and expresses the reward implicitly through the log-ratio between policy and reference probabilities. The result is a classification-style loss on preference pairs: -log sigmoid(beta * [(log pi(y_w|x) - log pi_ref(y_w|x)) - (log pi(y_l|x) - log pi_ref(y_l|x))]). It needs log-probabilities of both responses under the trained policy and a frozen reference (the latter can be precomputed), but no reward model and no sampling loop. Tradeoffs: it learns only from the fixed preference data, and it can over-optimize when that data is narrow.

Evaluating a fine-tune. Measure the target behavior on held-out data, plus regression suites for general ability, instruction following and safety behavior. Compare against the strongest prompted baseline, not only the untuned model.

Common misconceptions

• "Fine-tuning is how you teach a model your documents." It is unreliable for facts and stale as soon as documents change; use retrieval for knowledge.
• "LoRA adds inference latency." Not when merged; unmerged adapters add a small extra matrix product.
• "A higher LoRA rank is always better." It adds capacity and overfitting risk; small ranks are often enough, and the right value is an empirical question.
• "DPO makes preference-data quality less important." It is only as good as its pairs; noisy or narrow preferences give noisy or narrow behavior.
• "The KL penalty is a minor regularizer." It is what keeps the policy where the reward model remains valid.
• "RLHF mainly teaches the model new knowledge." It mostly shapes which behaviors the model expresses.

How to approach it

Be ready to argue the decision first: the failure, the baseline, the available data and the evaluation. Then learn the LoRA mechanics from the paper and library docs, and the RLHF and DPO pipelines from InstructGPT and DPO. If you have GPU access, run a small LoRA SFT and a DPO step on public data and evaluate before and after. In interviews, describe only what you actually built and measured.