Serving metrics: goodput, TTFT and inter-token latency

Why it matters

Throughput alone hides what users experience. LLM serving is judged by time to first token, the pace of streamed tokens, end-to-end latency and the rate of requests served within those targets, reported as percentiles at a stated load. Precise definitions and honest measurement separate a useful benchmark from a misleading one, and interviewers test both.

Core ideas

Definitions (state them, because tools differ).

• Time to first token (TTFT): from request submission to the first output token. It includes queueing and prefill and, when measured at the client, network and tokenization.
• Inter-token latency (ITL): the gap between consecutive streamed tokens. Time per output token (TPOT) is usually a request's average after the first token, (end-to-end latency − TTFT) / (output tokens − 1). Some tools include the first token or average over tokens rather than requests.
• End-to-end latency: TTFT plus generation time. It scales with output length, so compare at similar lengths or use per-token metrics.
• Throughput: requests per second, output tokens per second, or total tokens per second. Prefill-heavy and decode-heavy workloads give very different numbers on the same system.

Goodput. Following DistServe, goodput is the request rate a system sustains while a target fraction of requests meets every SLO, for example both a TTFT bound and a TPOT bound. Raw throughput can rise while goodput falls: a bigger batch produces more tokens per second but pushes TPOT past its bound.

Percentiles and tails. Report p50, p90 and p99 together with the number of requests behind them. Tails come from queueing, long prompts, preemption and stalls. A p99 computed from a few hundred requests rests on a handful of observations, so use enough requests and repeat runs.

Load generation. A closed-loop client (a fixed number of users, each waiting for its response) lowers its arrival rate when the server slows, so it never records the queueing delay that independent users would have seen; this measurement error is known as coordinated omission. An open-loop generator issues requests at a set rate, often Poisson or with tunable burstiness, independent of responses. Sweep the rate from light load to past saturation and plot latency percentiles against achieved throughput; the knee is where queueing grows sharply. It is the same latency-versus-load method used for key-value stores such as memcached.

Realistic workloads. Prompt and output length distributions, shared prefixes and burstiness drive results. Public traces such as the Azure LLM inference traces provide arrival times and token counts to replay.

Failures and fairness. Count rejected, timed-out and failed requests; dropping them from latency statistics makes overload look healthy. Report per-tenant or per-request-class percentiles, because one tenant's long prompts can inflate everyone's TTFT through head-of-line blocking.

Server-side breakdown and capacity. Engines expose queue time, prefill time, decode time and KV-cache usage; use them to explain client-side percentiles. Little's law (concurrency = arrival rate × time in system) connects a measured goodput curve to the number of replicas needed for a peak rate with headroom.

Common misconceptions

• "More tokens per second means a better system." Without latency SLOs and the load level, throughput can be bought with unacceptable TPOT; compare goodput under the same SLOs.
• "Average latency is enough." Averages hide bimodal behavior such as preemptions or cold caches, and users feel the tail.
• "N concurrent clients measure capacity." Closed loops hide queueing; use open-loop arrival rates for latency-under-load curves.
• "TTFT measures model speed." Under load it is mostly queueing; separate queue time from prefill time with server metrics.
• "Failed requests can be left out." Report them; otherwise overload looks like low latency.

How to approach it

Write a benchmark plan before running: model, hardware, engine version and settings, workload source, arrival process, rates to sweep, warm-up, duration, repetitions, metric definitions and percentiles, SLOs for goodput, and how failures are counted. Then present the latency-throughput curve with the knee marked. In interviews, define each metric before quoting any number.