Trace-driven evaluation and revisiting your cloud coursework

Why it matters

Cloud-systems claims are only as credible as the workload behind them. Trace-driven evaluation replays or models production behavior (arrival patterns, request sizes, resource usage) to test policies under realistic conditions. Research interviewers also ask about your own experiments: what you measured, how, and how sure you are. Re-running and documenting your cloud coursework with this discipline turns it into evidence you can defend.

Core ideas

Public traces. Google's cluster data covers one Borg cell from 2011 and eight cells from May 2019, the latter queried through BigQuery. The Azure Public Dataset includes VM traces, a packing trace, Azure Functions invocations (per-minute counts in 2019, per-invocation timing in 2021) and LLM inference traces with token counts. Alibaba's traces cover co-located online and batch workloads, GPU clusters and microarchitectural contention. Each accompanies a paper; read it before using the data.

Three ways to use a trace. Replay keeps exact arrivals and sizes but has no feedback: real users and retry logic react to latency, a replay does not. Synthetic workloads fitted to trace distributions allow scaling and what-if studies but can lose correlations and burstiness. Simulation reaches scale cheaply but must be validated against at least one real run.

Threats to validity. Time granularity (per-minute bins cannot reconstruct sub-minute inter-arrival times); sanitized or normalized units; sampled applications or regions; scaling a trace down to a small testbed, which changes burstiness and packing difficulty; open- versus closed-loop load; warm-up and steady state; and variance across machines and time in the cloud.

Latency-versus-load method. Use an open-loop generator, sweep offered load from light to past saturation, and at each point record achieved throughput (not only the target), latency percentiles, errors and utilization. Repeat runs, randomize the order of configurations, and report variability. Saturation is where achieved throughput stops tracking offered load or the tail grows sharply.

Revisiting your coursework: procedure, not results

Pick one experiment, such as memcached latency versus load with mcperf, an interference study, or a Kubernetes resource-allocation policy.

1. Setup. Machine or VM types, CPU model, core count, SMT on or off, kernel version, memcached version and flags (threads, memory), Kubernetes version, node layout, pod requests and limits, CPU pinning.
2. Load generator. Exact tool and commit (several different tools are called mcperf, so record the repository), agents, threads, connections, pipelining depth, key and value size distributions, read/write mix, QPS schedule, duration per point, warm-up, and where clients ran relative to the server.
3. Metrics. Which percentile defines the SLO and where it is measured, achieved versus target QPS, errors and timeouts, per-core utilization.
4. Repetitions. Re-run in randomized order and note which machines and times were used.
5. Interference. Each antagonist, the resource it stresses, its placement and intensity, and a full latency-load curve under each.
6. Kubernetes or allocation policies. What was co-located, cores per job, how decisions were made (static or controller-driven), the SLO, batch completion times and the scheduling event log.
7. Write-up. A one-page log and a two-minute explanation: question, setup, method, what the curves show, limitations, what you would change.

Questions to answer in writing: Which variables were fixed and which swept? How do you know the load generator was not the bottleneck? How large was run-to-run variance, and does it change the conclusion? What would differ on other hardware? Never fill in numbers you did not record; if results are lost, say so and explain how you would re-measure.

Common misconceptions

• "A replayed trace is ground truth." Replays lack feedback and traces are sampled and sanitized; state what the trace cannot capture.
• "Target QPS equals achieved QPS." Generators fall behind near saturation; plot achieved throughput.
• "One run per point is enough if the curve looks smooth." Cloud environments vary across machines and time; repeat and report variability.
• "Closed-loop clients measure latency under load." They slow down with the server and hide queueing.
• "Scaling a trace down preserves its behavior." Compression and down-scaling change burstiness and packing difficulty; test sensitivity.

How to approach it

For a public trace, reproduce one figure from its paper before asking new questions. For your experiment, complete the log before re-running anything. In interviews, lead with the question and method, show the curve, and volunteer the limitations.