Throughput, latency, Little's law and Amdahl's law

Why it matters

Capacity questions such as "how many workers do we need?", "will adding machines fix this?" or "why did latency triple at 80 percent load?" come up in system-design, SRE and research interviews, and they reward a few laws used carefully. The laws are simple; the skill lies in knowing which quantity is an average, which boundary you are measuring and which assumption a real system violates. The same reasoning sizes LLM serving: the number of sequences in flight equals request rate times average request duration, which sets the batch size and KV-cache memory the server must sustain.

Core ideas

Vocabulary. Throughput is completions per unit time. Latency is time per request, best described as a distribution (p50, p99, maximum). Utilization is the fraction of time a resource is busy, and concurrency is the number of requests in progress. For a single resource, the utilization law ties them together: utilization = throughput × service time per request.

Little's law: L = λW. The average number of items in a system equals the average arrival rate times the average time each item spends inside. Example: 400 requests per second at 250 ms average latency means about 100 requests in flight, so a synchronous design needs roughly 100 workers or connections plus headroom. Its assumptions and limits:

• It relates long-run averages of a system in steady state: arrivals and departures balance and the backlog is not growing. During overload or a traffic spike it does not describe the transient.
• L, λ and W must describe the same boundary. Measuring W at the server but L at the client, which also counts time queued in a load balancer, breaks it.
• It holds for any arrival distribution and service order, but it says nothing about percentiles; you cannot derive p99 from it.

Queueing near saturation. For the simplest single-server model (random arrivals, exponential service times), mean response time is S / (1 − ρ), where S is the service time and ρ the utilization: twice the service time at 50 percent, ten times at 90 percent. Real systems are not that model, but the shape generalizes: waiting grows sharply as utilization approaches one, and more variable arrivals or service times make it worse. That is why capacity plans target headroom rather than full utilization.

Bottlenecks. If each request needs D_i seconds of resource i, throughput cannot exceed 1 / max D_i, and latency cannot fall below the sum of the D_i. A single database that needs 4 ms per request caps throughput at 250 requests per second no matter how many application servers you add.

Amdahl's law. If a fraction p of the work parallelizes perfectly across n processors, speedup = 1 / ((1 − p) + p / n), which never exceeds 1 / (1 − p). With p = 0.95, eight cores give about 5.9× and unlimited cores 20×. What it assumes or ignores:

• Fixed problem size (strong scaling). Gustafson's law covers work that grows with n, where scaled speedup can stay close to linear.
• No parallel overhead. Real systems pay for synchronization, communication and contention, and can slow down beyond some n; the Universal Scalability Law adds contention and coherency terms to model that.
• Perfect balance and identical processors. Stragglers and heterogeneous cores change the result.
• No memory effects. Splitting data can make it fit in cache (superlinear speedup) or saturate memory bandwidth (flat speedup).

Critical path. Latency is set by the longest chain of dependent steps (the span), and total work divided by span bounds useful parallelism. Optimizing work off the critical path saves resources, not latency. With fan-out to many backends, the slowest backend sets the response time, so tail latency compounds.

Common misconceptions

• "Little's law gives me p99." It is a statement about means.
• "Little's law needs Poisson arrivals or FIFO service." It is distribution-free; it needs steady state and a consistent boundary.
• "Amdahl proves parallelism is futile." It bounds fixed-size problems with a fixed serial fraction; bigger problems or a smaller serial part change the answer.
• "Doubling servers halves latency." It reduces queueing; service time and the critical path are unchanged.
• "Running at 95 percent utilization is efficient." It buys throughput with large and volatile queueing delay.
• "Average latency describes user experience." Users and fan-out callers experience the tail.

How to approach it

Practice converting between rate, latency and concurrency until it is automatic, and always name the assumption you are relying on. Given scaling data, ask whether the curve flattens (Amdahl-shaped) or peaks and falls (USL-shaped). In interviews, write the formula, plug in numbers, then name the assumption most likely to be violated in that scenario.