Proxies, load balancing and CDNs

Why it matters

It is easy to say "put a load balancer in front" in a design discussion. Good answers say which kind, where TLS terminates, how a backend is chosen, how failure is detected and what happens to long-lived connections during a deploy. Many incidents come from exactly these details: uneven load from connection-level balancing of HTTP/2 or gRPC, health checks that pass while the service is broken, retries multiplied by every proxy layer, or a CDN cache serving stale or private data.

Core ideas

Forward versus reverse proxy. A forward proxy acts for clients (corporate egress, filtering, caching). A reverse proxy acts for servers: it terminates client connections and opens its own connections to backends, which lets it terminate TLS, route by host or path, add headers, buffer slow clients, enforce limits and cache responses.

L4 versus L7. An L4 load balancer works with transport information: it chooses a backend per connection (or per UDP flow) from the five-tuple and forwards packets, sometimes without terminating TCP at all. It is fast and protocol-agnostic but cannot see requests, so one long-lived HTTP/2 or gRPC connection carrying thousands of requests lands on a single backend. An L7 load balancer terminates the connection, parses HTTP and balances each request; it can route by path or header, retry and observe status codes and latency, at the cost of more CPU and of becoming part of the protocol (timeouts, header limits, version translation). Large deployments layer them: DNS or anycast spreads clients across sites, L4 spreads connections across a fleet of L7 proxies, and L7 spreads requests across services.

Choosing a backend. Round robin ignores load. Least-outstanding-requests and "power of two random choices" (pick two at random, use the less loaded one) adapt cheaply to slow backends; weighted variants handle mixed capacity. Hashing keeps a client or key on the same backend for cache locality; consistent hashing and Maglev-style lookup tables remap only a small fraction of keys when backends change, and let a fleet of stateless L4 balancers agree on where a connection belongs.

Health and failure. Active health checks probe an endpoint; passive checks (outlier detection) eject backends that return errors or time out. A check that only confirms "process alive" misses broken dependencies, while one that tests every dependency can eject the whole fleet when a shared dependency blips, so cap the fraction that may be ejected. Connection draining lets in-flight requests finish before a backend leaves. Retries at several layers multiply load; retry budgets and deadlines keep them bounded.

CDNs. A CDN is a geographically distributed set of reverse proxies that cache content near users, terminate TLS close to them (saving handshake round trips) and keep warm connections back to the origin. Clients reach an edge through DNS or anycast. HTTP caching headers (Cache-Control, Vary, validators such as ETag) decide what may be stored; cache keys and invalidation are where mistakes leak private data or serve stale content.

Common misconceptions

• "An L4 balancer spreads requests evenly." It spreads connections; with long-lived multiplexed connections the request load can be badly skewed.
• "Round robin is good enough." With uneven request costs or a slow backend, load-aware choices noticeably reduce tail latency.
• "Health checks make failover automatic and safe." Shallow checks miss real failures and deep checks can eject everything at once; design for both.
• "A CDN only helps static files." Terminating TLS near users and reusing origin connections also speeds up uncacheable requests.
• "The proxy is transparent." Every proxy adds timeouts, buffering, size limits and its own retry policy, and hides the client address unless it forwards it (X-Forwarded-For, PROXY protocol).

How to approach it

For any design, list the layers in order (DNS or anycast, L4, L7, service) and what each decides. Practice explaining connection versus request balancing with a gRPC example, and how consistent hashing limits remapping. Senior-level discussions add deploy safety (draining, slow start for new backends), failure domains, overload behavior and the interaction of retries across layers.