Statistical uncertainty, seeds and paired comparisons

Why it matters

"Did you run it more than once?" is a standard question, and "what is the variance across seeds?" follows quickly for ML work. Small improvements are routinely within run-to-run noise. In industry the same reasoning decides whether a regression or an A/B result is real.

Core ideas

Where variance comes from. In systems: thread scheduling, cache and TLB state, memory placement, CPU frequency and turbo, interrupts, background daemons, network jitter and co-located tenants. In ML: initialization, data order, dropout, nondeterministic GPU kernels and the data split itself. A seed controls only the randomness routed through it; fixing one seed makes one sample repeatable and hides seed-to-seed variance instead of measuring it.

The experimental unit. Ask what could be repeated independently. Ten thousand request latencies from one run are not ten thousand independent samples: they share warm caches, process layout and machine state, and queueing correlates neighbouring requests. The unit is usually the run (fresh process, ideally fresh placement), the seed, the machine or the data split. Treating correlated measurements as independent, sometimes called pseudo-replication, yields intervals that are far too narrow. Runs are not automatically independent either: consecutive runs on one shared machine or cloud cluster see the same neighbours, thermal state and time-of-day load. Spread runs over time and machines, or block by machine and compare variants within each block. Variance can sit at several levels (builds, machines, process restarts, iterations); vary every level your claim should cover.

How many runs. Start with five to ten, inspect the spread, then decide; interval width shrinks roughly in proportion to one over the square root of n, so quadrupling the runs about halves it. Tail metrics such as p99 need more and longer runs than means. Exclude warm-up explicitly and say how.

Confidence intervals. For a mean over n independent runs, a t-interval (mean plus or minus a t quantile times the standard error) is reasonable unless run-level values are strongly skewed. For medians, percentiles and ratios, use bootstrap or quantile-based intervals. Compute one statistic per run (that run's p99) and then the interval across runs; pooling every request into one p99 hides between-run variance.

Pair and interleave. When variants A and B use the same seeds, traces and machines, analyse per-pair differences: shared noise cancels, so the interval of the difference is often much narrower than two separate intervals suggest. Interleave runs (A, B, B, A, ...) or randomize order so environmental drift (thermal state, time of day, a noisy neighbour) does not line up with the treatment.

# 95% percentile-bootstrap interval for the mean paired difference (B - A)
import random, statistics
def paired_bootstrap_ci(a, b, iters=10_000):
    d = [y - x for x, y in zip(a, b)]          # one value per paired run
    m = sorted(statistics.mean(random.choices(d, k=len(d))) for _ in range(iters))
    return m[int(0.025 * iters)], m[int(0.975 * iters) - 1]

When not to trust a p-value. A p-value measures how surprising the data would be under a specific null model; it is not the probability that your hypothesis is true and says nothing about effect size. It misleads when you test many metrics or configurations and report the best, stop collecting once significance appears, ignore dependence between samples, or have so many samples that a negligible difference becomes "significant". A non-significant result from three runs is not evidence of no effect. Prefer intervals on effect sizes, an analysis plan fixed in advance and a stated minimum effect that matters.

Reporting. State n, what one sample is, the interval method and level, and show distributions (CDFs, box plots) rather than single bars. Report all seeds or runs, not the best.

Common misconceptions

• "Ten thousand requests give a tight interval." Not if they come from one run; the run is the unit.
• "A fixed seed solves variance." It makes one sample repeatable; you still need several seeds to know the spread.
• "Overlapping error bars mean no difference." With paired designs the per-pair difference can be clearly non-zero while separate intervals overlap.
• "p < 0.05 means the improvement is real and important." It may reflect many attempts, dependence or a negligible effect; check the interval against the practical threshold.
• "Standard deviation bars and confidence intervals are interchangeable." They answer different questions; always say which you plot.

How to approach it

Re-run one experiment you control ten times, compute per-run statistics and a paired interval against a variant, and state the result in one sentence with n, unit and interval. Graduates are expected to insist on repeated runs and name the unit; senior candidates should raise dependence, tail metrics and multiple comparisons unprompted.