Profiling, bottlenecks and benchmarking pitfalls

Why it matters

"The service got slow, what do you do?" is a common prompt for SRE, infrastructure and performance-sensitive roles, and the weak answer is a list of tools. The strong answer is a method that finds the limiting resource quickly and verifies the fix with a measurement you trust. The second half matters as much: a reported speedup, including one in a paper, can come from a benchmark that measured a warm cache, a throttled CPU, the load generator itself, or an average that hid the tail.

Core ideas

Method first. State the metric, such as p99 latency at a given load or throughput within an SLO, and reproduce the problem. For each resource (CPUs, memory, disks, network, locks, thread pools) check utilization, saturation and errors, the USE method, before reading code. Then change one thing at a time and re-measure under the same conditions.

Sampling, instrumentation, tracing.

• Sampling profilers such as perf record, py-spy or async-profiler interrupt at a fixed rate and record stacks. Overhead is low enough for production, but results are statistical and depend on stack walking: missing frame pointers or symbols give broken stacks.
• Instrumentation such as cProfile, manual timers or tracing spans records every call or region. Counts are exact, but overhead distorts short functions.
• Tracing with tracepoints or eBPF tools records specific events (system calls, disk I/O, scheduler switches) with timestamps, which is how you attribute latency rather than CPU time.

Counters. perf stat reports cycles, instructions, cache and branch misses, context switches and page faults. Low instructions per cycle with many cache misses points to memory stalls rather than "slow code". High CPU utilization can hide spinning or stalls, so utilization alone does not prove the work is useful.

Flame graphs. Each box is a function in the sampled stacks; its width is the share of samples containing it, and callees sit on top of callers. The x-axis is sorted alphabetically, not by time, so look for wide plateaus rather than a left-to-right sequence. A CPU flame graph shows only on-CPU time; latency spent waiting on locks, I/O or the run queue needs off-CPU analysis, which records where threads block and for how long.

Classifying bottlenecks.

• CPU-bound: busy cores and a wide hot path; improve the algorithm or parallelize.
• Memory-bound: low IPC and many cache misses; change layout or access pattern.
• I/O-bound: high device utilization or queue depth and threads blocked in I/O; batch, cache or change the access pattern.
• Lock-bound: low CPU and low throughput with threads waiting in futex calls; shrink critical sections or shard.
• Scheduler-bound: long run-queue latency or, in containers, quota throttling recorded in cpu.stat.

Measuring latency. Record full distributions in histograms, and never average percentiles across machines; merge histograms instead. Many load generators are closed loops: each connection waits for a response before sending its next request, so when the server stalls they stop sending and never record the requests that would have waited. This coordinated omission hides exactly the bad periods. Use constant-rate, open-loop generators such as wrk2, or correct each latency to the request's intended send time.

Benchmarking pitfalls. Missing warmup (JIT compilation, lazy initialization, connection pools, page cache); unstated cache state; CPU frequency scaling, turbo and thermal throttling; noisy neighbors on shared hosts; a client, network or logging bottleneck masquerading as the system under test; compilers deleting code whose result is unused; data sets that fit in cache unlike production; too few runs and no variance. Even link order and environment size can shift results, as Mytkowicz et al. showed in "Producing Wrong Data Without Doing Anything Obviously Wrong!" (ASPLOS 2009). Active benchmarking means profiling the benchmark while it runs to confirm what limits it.

Common misconceptions

• "The flame graph x-axis is time." It is an alphabetical merge of stacks; flame charts show time order.
• "100 percent CPU means we need faster CPUs." It may be spinning, garbage collection or memory stalls; check IPC and what the cycles are doing.
• "Low CPU means the service has headroom." Threads may be blocked on locks or I/O, or throttled by a CPU quota.
• "The profiler shows why requests are slow." CPU profilers miss waiting; latency needs off-CPU analysis or tracing.
• "Averaging each host's p99 gives the fleet p99." Percentiles do not average; merge the distributions.
• "A microbenchmark speedup carries over to production." Cache state, data shape and contention differ.

How to approach it

Practice the method aloud on a scenario until it takes five minutes. Then do it for real: profile a deliberately slow program with perf (or py-spy for Python), produce a flame graph, fix the hot spot and verify. In interviews, name the metric, the resource checks, one profiling step and how you will confirm the fix.