GPU architecture, memory hierarchy and arithmetic intensity

Why it matters

Most AI-infrastructure questions reduce to one diagnosis: is the GPU waiting on arithmetic, on memory, or on the host? Interviewers listen for whether you make that diagnosis before proposing a fix. Suggesting a faster GPU for a bandwidth-bound decode step, or kernel fusion for a step dominated by data loading, shows the model is missing. The same reasoning returns in training memory, FlashAttention, KV caches and quantization.

Core ideas

Execution model. A GPU contains many streaming multiprocessors (SMs). A kernel launch creates a grid of thread blocks; each block runs on one SM, and its threads execute in groups of 32 called warps that issue one instruction at a time for all their threads. When a warp waits on memory, the SM switches to another resident warp almost for free, so GPUs hide latency with parallelism rather than large per-thread caches. If few warps fit on an SM (many registers per thread, much shared memory per block), occupancy drops and latency shows. Enough parallelism to cover latency is the goal, not maximum occupancy.

Tensor cores run small matrix multiply-accumulate operations far faster than the general-purpose cores, but only for supported formats (16-bit floats and, on newer parts, TF32, int8 or fp8) and well-aligned shapes. Whether a plain fp32 matmul uses TF32 is a framework setting: check it.

Memory hierarchy. From fastest and smallest: registers (per thread), shared memory and L1 (per SM; the FlashAttention paper calls it SRAM), L2 (shared by all SMs), HBM (device memory, tens of gigabytes), and host memory over PCIe or NVLink. Each step down costs a large factor in bandwidth or latency. Global loads are efficient only when a warp's threads touch contiguous addresses (coalescing).

Arithmetic intensity and the roofline. Intensity I is FLOPs performed divided by bytes moved to or from HBM. With peak compute P and bandwidth B, attainable throughput is at most min(P, I × B). The ridge point P/B separates memory-bound from compute-bound work; compute it from your GPU's datasheet for the precision you use. Two reference cases in 16-bit:

• Elementwise op on N values: about 2N bytes read, 2N written, a few FLOPs per element, so I is around one FLOP per byte or less. It is memory-bound everywhere; its time is roughly bytes divided by bandwidth.
• Matmul (M×K)·(K×N): 2MNK FLOPs over at least 2(MK + KN + MN) bytes. For square n×n matrices I ≈ n/3, so large matmuls are compute-bound. A matrix-vector product has I ≈ 1 because each weight is read once and used once; batching b vectors raises I to roughly b. This is why small-batch LLM decoding is bandwidth-bound.

Overhead. Every kernel launch has a fixed CPU and driver cost that does not shrink with tensor size. With small tensors the GPU finishes before the host enqueues the next kernel, leaving gaps on the timeline. Remedies: larger batches, fusion (torch.compile, Triton or library kernels), CUDA graphs that replay a recorded launch sequence, and removing host synchronization (.item(), printing tensors, data-dependent shapes) from the hot loop.

Asynchrony. CUDA work is queued on streams and Python returns immediately, so a wall clock without synchronization measures enqueue time. Use CUDA events or torch.utils.benchmark, warm up, and synchronize before reading the clock.

Profilers. The PyTorch profiler gives per-operator time, memory and an exportable timeline. Nsight Systems puts CPU threads, CUDA API calls, kernels, copies and communication on one timeline ("where did the time go?"). Nsight Compute dissects one kernel: achieved bandwidth and FLOP/s, occupancy, stall reasons ("why is this kernel slow?").

Common misconceptions

• "nvidia-smi shows 100% utilization, so the GPU is saturated." That metric is the fraction of time at least one kernel was running; a tiny kernel running continuously reports 100%. Measure achieved FLOP/s or bandwidth instead.
• "Fewer FLOPs means faster." Only for compute-bound work. Memory-bound operations speed up when bytes moved go down, even if FLOPs go up; FlashAttention's recomputation is the standard example.
• "GPUs are fast because of big caches." Per-thread cache capacity is small; GPUs depend on many independent warps and coalesced access.
• "I timed it with time.time(), so the kernel takes 2 ms." Without warm-up and synchronization you measured enqueue time or one-time compilation and allocation.
• "Tensor cores are used for every matmul." They need supported dtypes and aligned dimensions; badly shaped matrices waste part of every tile.

How to approach it

Build a two-number habit: for any operation write down FLOPs and bytes, compute intensity, compare with the device's ridge point, then confirm with a measurement. At graduate level, expect to explain compute- versus memory-bound with matmul and elementwise examples. At mid level, expect to read a trace and rank fixes by payoff. At senior level, expect to set utilization targets for a whole system and judge when custom kernels are worth their maintenance cost.