Interpretability methods (optional)

Why it matters

Interpretability is a research area in its own right and a growing part of safety work. Even outside it, interviewers may ask how you would check whether a model "knows" something or why it made a prediction. A good answer depends on knowing which methods give correlational evidence and which support causal claims.

Core ideas

Probing trains a simple classifier on frozen activations to predict a property such as part of speech or a board state. High accuracy shows the information is decodable, not that the model uses it, and a flexible probe can learn the task by itself. Control tasks, which assign random labels with the same structure, measure how much accuracy comes from the probe's own capacity.

Attribution assigns importance scores to inputs: raw gradients, gradient times input, integrated gradients (which accumulates gradients along a path from a baseline input and satisfies a completeness property) and perturbation methods. Randomization tests have shown that some saliency maps barely change when the model's weights are randomized, so they reflect the input more than the model. The choice of baseline input also changes the answer.

Mechanistic methods try to reverse-engineer the computation. Activation patching replaces activations from one run with those from another, for example a corrupted prompt, and measures the effect on the output, which gives causal evidence about which components matter. Circuit analyses trace how specific attention heads and MLP layers implement behaviours such as induction: copying the token that followed an earlier occurrence of the current token. Because networks represent more features than they have neurons (superposition), single neurons are often polysemantic, and sparse autoencoders or dictionary learning are used to extract more interpretable feature directions.

Common misconceptions

• "High probe accuracy means the model uses that information." It shows decodability; interventions are needed to show use.
• "Attention weights explain the prediction." They show mixing weights in one layer; other components transform the result, and different weights can produce the same output.
• "A clean saliency map validates the method." Visual plausibility is not faithfulness; run randomization and perturbation tests.
• "A circuit found on a curated prompt set explains the behaviour in general." Check coverage on broader inputs and use mean or resample ablations rather than zeroing activations, which can push the model off its normal operating range.

How to approach it

Start with the Transformer Circuits framework paper and a TransformerLens tutorial on a small model, reproduce induction-head detection, then train a probe with a control task. Read the saliency sanity-check paper before trusting any attribution figure. In interviews, label every interpretability claim by its evidence type: decodability, attribution or intervention.