Database workload evaluation

Why it matters

Many database papers report large speedups that shrink or reverse under a different workload. Reviewers and research supervisors look first at the evaluation: does the workload exercise the claimed mechanism, are baselines tuned, and do conclusions hold across the regimes that matter? Being able to plan such an evaluation, and to critique one, is central to a database research internship or thesis.

Core ideas

From hypothesis to workload. A claim like "our protocol scales better under contention" names its variable: contention. The workload must sweep it (for example the Zipf parameter or the number of hot rows) while holding other factors fixed, and should include a regime where the claim is not expected to hold, as a sanity check. Avoid setups where an unrelated bottleneck, such as the client, the network or logging, decides the result.

What standard benchmarks capture and miss. TPC-C has realistic transaction mixes and contention on a few rows, but fixed access patterns and scaling rules. TPC-H has well-understood analytical queries, but data with little skew or correlation. YCSB isolates key-value operations with tunable distributions but has no multi-key transactions. JOB brings real correlations for join ordering but is read-only and uses one dataset. Choose benchmarks by the mechanism under test, and add a real or trace-derived workload when external validity matters.

Parameters that flip conclusions. Concurrency-control rankings change with contention and with the share of multi-partition transactions; storage-engine rankings change with read/write ratio, value size and whether data fits in memory; optimizer results change with correlations and physical design. Published evaluations such as Yu et al. on many-core concurrency control and Harding et al. on distributed protocols show these crossovers, which is exactly what a regime map should report.

Baselines and fairness. Compare against the strongest relevant baseline, configured with comparable effort, at the same durability and isolation guarantees. Implementing all contenders in one framework, as several concurrency-control studies do, makes comparisons fairer, but the results then describe the framework rather than production systems.

Measurement and reporting. Run to steady state, repeat, and report distributions (median and tail, with spread across runs). State hardware, versions, configuration and generator seeds. Check correctness on every run (invariants, result checksums). Publish scripts and raw data so a reader can regenerate a figure.

Common misconceptions

• "A standard benchmark makes an evaluation valid." Validity depends on whether the benchmark exercises the claim; a standard suite can miss the relevant variable entirely.
• "A large speedup on one workload is a general result." Without a sensitivity analysis it is one point in the parameter space.
• "Synthetic workloads are worthless." They are the best tool for isolating a mechanism; complement them with realistic workloads rather than replacing them.
• "Sharing code makes work reproducible." Reproduction also needs configurations, seeded generators, environment details and raw measurements.
• "Matching default configurations is fair." Defaults encode different durability, memory and parallelism choices.

How to approach it

Take one published evaluation, for example Wu et al.'s MVCC study or the evaluation section of the Spanner paper, and write a one-page critique: question, workload, baselines, metrics, missing regimes. Then plan a small study of your own for a mechanism you care about, with a parameter sweep and a correctness check, and run a reduced version with pgbench, YCSB or BenchBase. In research interviews, use both as evidence of evaluation thinking and describe only what you actually ran.