Benchmark realism and external validity

Why it matters

Reviewers and interviewers rarely doubt that your numbers were measured; they ask whether the numbers mean anything outside your setup. A cache that wins on uniform keys may lose on skewed ones, a scheduler tuned on a synthetic arrival process may fail on bursty production traffic, and a model that tops a public benchmark may have seen its test items during training.

Core ideas

Three validities. Internal validity: did your change cause the difference, or did something that changed with it? Construct validity: does the metric capture the property you claim (is saturation throughput the right metric for an interactive service)? External validity: does the result hold for other workloads, machines, scales and datasets? Each claim should say which settings it covers.

Workload choices. Synthetic workloads give control: you can sweep key skew (for example Zipf parameters), request sizes, read/write mix and burstiness, and isolate effects. The danger is choosing parameters that favour your design. Trace-driven evaluation adds realism (diurnal patterns, heavy tails, correlated arrivals), but traces are often old, sampled or anonymized, come from another environment, and change meaning when replayed against a system that responds differently from the original. Production experiments are the most realistic and least controllable. Strong evaluations combine them: traces to show relevance, synthetic sweeps to explain behaviour.

Open versus closed loops. A closed-loop generator sends the next request only after a response, so a slow system automatically receives less load and its tail latency is under-reported. An open-loop generator sends requests on a schedule regardless of responses, which matches many internet-facing services. Schroeder, Wierman and Harchol-Balter show that the two models can lead to different conclusions about the same scheduling policy; state which one you used and why.

Hidden confounders. Mytkowicz et al. showed that innocuous details, the size of the UNIX environment and the link order of object files, shift memory layout enough to change conclusions about compiler optimizations; they recommend randomizing the setup and using causal analysis. Similar confounders include frequency scaling and turbo, SMT, NUMA placement, warm page caches and buffer pools, background services, cloud instance hardware generations and noisy neighbours. A factor that can change the outcome and was neither varied nor fixed limits your claim.

Cloud-specific pitfalls. Schwarzkopf, Murray and Hand describe seven sins: unnecessary distributed parallelism (no single-machine baseline), assuming performance homogeneity, picking low-hanging fruit as a baseline, forcing an abstraction onto a poorly matched problem, unrepresentative workloads, assuming perfect elasticity and ignoring fault tolerance. Heiser's list of benchmarking crimes adds selective benchmarking, improper baselines, misleading summary statistics and missing platform details.

ML and LLM benchmarks. Test items can leak into training data; small prompt changes can reorder models; heavily used benchmarks saturate and reward overfitting to their quirks; automated judges have biases of their own. Agent benchmarks add another layer: tasks may not resemble real tool environments, and scores can depend on harness details.

Common misconceptions

• "A standard benchmark guarantees comparability." Only with the same version, configuration, subset and environment; unexplained subsetting is a classic crime.
• "Traces are ground truth." They are evidence from one environment and period, shaped by the system that produced them.
• "More load-generator clients fix tail measurement." A closed-loop design still under-reports tails; the loop model matters more than client count.
• "Stable results across runs mean an unbiased setup." Stable but biased is common; vary the setup itself to expose bias.
• "Topping a leaderboard shows general improvement." It shows improvement on that benchmark, which may be contaminated or saturated.

How to approach it

For every claim, write "this holds for workloads like ..., on hardware like ..., at scales up to ..." and check that the evaluation covers it. Audit one published evaluation against the seven sins and Heiser's list, then try the setup-perturbation exercise. Graduates should spot a closed-loop tail measurement or a missing single-machine baseline; senior candidates should design workloads that reveal where a system fails.