Quantization, kernels and quality tradeoffs

Why it matters

Quantization is a standard cost lever in serving: fewer bits per weight, activation or cached token means less memory to hold and less data to move. Whether it speeds anything up depends on which resource is the bottleneck and whether fast kernels exist for the format on your hardware, and it can quietly reduce quality. Interviewers listen for whether you name what is quantized, how it was calibrated, and when it will not help.

Core ideas

Mapping. Affine quantization stores q = round(x / scale) + zero_point, clamped to the integer range, and reconstructs x̂ = scale × (q − zero_point); symmetric schemes fix the zero point at 0. One scale can cover a whole tensor, one output channel, or a small group of consecutive weights; finer granularity lowers error but stores more scales. Low-bit floating formats such as fp8 keep an exponent, which suits wide value ranges, but still need scaling factors.

Weight-only quantization (for example 4- or 8-bit weights with 16-bit activations) shrinks the model and the bytes read per decode step. Kernels dequantize weights in registers and multiply in 16-bit, so arithmetic is unchanged or slightly higher. It helps bandwidth-bound regimes such as small-batch decoding and does little for compute-bound prefill or large batches.

Weight-and-activation quantization (8-bit integers or fp8 for both operands) can use low-precision tensor-core paths with higher peak throughput, which helps compute-bound work, but only on hardware and kernels that support the format.

Outliers and calibration. LLM activations contain a few channels with very large magnitudes, so a single per-tensor scale wastes most of the integer range on them. LLM.int8() keeps outlier dimensions in 16-bit and quantizes the rest; SmoothQuant rescales channels offline so activations become easier to quantize and the weights absorb the difficulty. Post-training methods use a small calibration set: GPTQ quantizes weights layer by layer and adjusts the remaining weights to compensate using approximate second-order information; AWQ uses activation statistics to protect the weight channels that matter most through per-channel scaling.

KV-cache quantization cuts bytes per cached token, raising concurrency and lowering decode bandwidth, with its own accuracy risks at long contexts.

When quantization does not speed things up: compute-bound phases with weight-only formats; formats without an optimized kernel on your GPU, which fall back to dequantizing and multiplying in 16-bit; small models where launch overhead dominates; batch sizes where dequantization costs more than the bandwidth saved; and stacks whose bottleneck is scheduling, sampling or the CPU.

Common misconceptions

• "4-bit weights make inference four times faster." The ceiling is set by bytes saved in bandwidth-bound phases and by kernel quality; compute-bound phases may not speed up at all.
• "If perplexity barely moves, quality is preserved." Perplexity can hide losses on reasoning, long-context or domain-specific tasks; evaluate on the tasks you care about.
• "int8 means integer arithmetic throughout." Weight-only int8 dequantizes and computes in 16-bit; only weight-and-activation paths use integer tensor cores.
• "Calibration data does not matter." Scales and compensation are fitted to calibration activations, so a mismatched set skews them.

How to approach it

Measure two regimes, decode at batch size 1 and a large batch or long prefill, for a 16-bit model and a quantized version on the same engine and GPU, and add a small task-quality comparison. In interviews, start from the bottleneck (bandwidth versus compute), then the format and kernel path, then the quality evaluation.