Alignment and safety evaluation research (optional)

Why it matters

Alignment and safety teams spend much of their effort on evaluation: deciding whether a model behaves as intended, under which conditions it fails, and whether a training change helped. Many evaluation results are fragile because the metric does not measure the intended property or the grader is biased. Interviewers for these roles probe whether you can design an evaluation whose conclusion you would trust.

Core ideas

Reward hacking. When an optimizer maximizes a proxy, it exploits gaps between the proxy and the goal: specification gaming in RL environments, or verbose, flattering answers after RLHF. Studies of learned reward models find that the true objective first improves and then degrades as the policy moves further from its starting point, even while the proxy reward keeps rising. Mitigations include KL penalties, reward-model ensembles, held-out gold evaluations and early stopping.

Preference data. Pairwise preferences from humans or models are noisy and biased: annotators disagree, often prefer longer or more confident answers, and apply guidelines unevenly. These biases become part of the reward model and of policies trained with RLHF or DPO. Measure agreement and audit simple correlates such as response length before trusting a preference dataset.

Red teaming. Manual red teaming finds failure modes creatively; model-assisted red teaming generates many candidate attacks, which then need deduplication, diversity measurement and severity grading. Record prompts, model versions and sampling settings so findings reproduce.

Validity. An evaluation has construct validity when its score tracks the property in the claim, for example "refuses dangerous requests" rather than "matches a refusal template". Threats include contamination, saturated benchmarks, sensitivity to prompt format, small item counts and grader bias. Model graders can show position bias, verbosity bias and preference for their own outputs; validate them against human labels on a sample, swap answer order, and report agreement. Report uncertainty, since many safety evaluations have few items.

Common misconceptions

• "A higher reward-model score means a safer or better model." Past some point the proxy and the true objective diverge; check against held-out human or gold evaluations.
• "An LLM judge is an objective grader." Judges have measurable biases and need calibration against humans and order-swapping checks.
• "Passing a refusal benchmark shows the model is safe." It shows behaviour on that distribution and format; adaptive attacks and paraphrases often find failures.
• "More red-team prompts means better coverage." Without diversity and severity analysis, many automated attacks are near-duplicates of one failure.

How to approach it

Read the overview papers, then build a small evaluation with an evaluation harness: define the behaviour, write items, validate the grader against your own labels, sample several times per item and report intervals. In interviews, state the threat model and the validity argument before the metric.