Experimental design: controls, baselines and ablations

Why it matters

"How would you evaluate that?" follows almost every research idea in an interview, and a weak evaluation is a frequent reason good ideas are not believed. The same discipline governs A/B tests, performance-regression hunts and model comparisons in industry.

Core ideas

Name the variables. Factors are what you vary deliberately (system variant, load level, model size). Levels are the values you try. Response variables are the metrics (p99 latency, throughput per core, accuracy, cost per task). Everything else is held fixed and recorded, or randomized when it cannot be fixed (run order, placement, seeds). Le Boudec's methodology chapter shows how a hidden factor that varies along with your treatment can reverse a conclusion, as in Simpson's paradox.

Controls. A negative control is a change that should have no effect, such as comparing a configuration with itself ("A versus A"); a visible difference means the harness or environment is noisy or biased. A positive control is a known effect that should reappear, such as reproducing a published baseline result on your setup before measuring your variant.

Baselines. Include at least three kinds where possible: a simple baseline (default configuration, a bigger cache, a longer timeout, plain keyword retrieval), the strongest published alternative, and a bound such as an oracle or a theoretical limit. Fairness means the same hardware, software versions, data, workload and, crucially, the same tuning budget. If your method received fifty hyperparameter trials, so does the baseline, or you report results as a function of budget as Dodge et al. propose. Melis et al. found that carefully re-tuned standard language-model baselines compared favourably with newer architectures, the classic warning on this point.

Ablations. Remove one component at a time and measure the drop; adding components one at a time shows cumulative effects. If two components only help together, a small factorial design (two levels per factor, all combinations) quantifies the interaction. Without ablations you cannot say which part of the design earns its complexity, one of the failures Lipton and Steinhardt describe in ML papers.

Sensitivity analysis. Every design has parameters you picked (thresholds, window sizes, timeouts, learning rates, prompt templates) and workload assumptions (key skew, request sizes, burstiness). Sweep them and report where the method stops winning. A method that wins only near its defaults is fragile, and reviewers know it.

Error analysis. Aggregates hide structure. Group failures, slowest requests or misclassified examples by cause, input size, tenant, query type or tool, and report per-slice results. This prevents claiming an average gain that hides a regression for an important slice, and it often suggests the next hypothesis.

Plan before running. Write the hypothesis, metrics, comparisons, number of runs and analysis in a short plan. Hoefler and Belli's rules for reporting performance (state the base case, summarize appropriately, give intervals for nondeterministic data, document every factor) double as a checklist.

Common misconceptions

• "Quoting the paper's numbers is a baseline." Numbers from another machine, version or data split are not a controlled comparison; re-run the baseline in your setup.
• "An untuned baseline is conservative." It inflates your gain. Equal tuning effort, or results reported against budget, is the fair comparison.
• "Ablations are optional if the full system wins." Without them you cannot claim that any specific component matters.
• "One factor at a time is always correct." It misses interactions; use a small factorial design when components may depend on each other.
• "Averages are enough." Means over mixed workloads hide regressions on slices that matter; report distributions and per-slice results.

How to approach it

Write a one-page evaluation plan for one of your hypotheses: factors and levels, baselines with tuning budgets, ablations, sensitivity sweeps, controls and the summary statistic for each metric. Then write down which result would make you abandon the idea. Graduates should propose a fair baseline and an ablation without prompting; senior candidates should name threats to validity and say which experiments they would cut first when compute is limited.