Flow control, congestion control and queueing

Why it matters

"Throughput is low although the link is fast", "latency spikes when someone uploads a file", "transfers between regions are slow": these are flow-control, congestion-control or queueing problems. Infrastructure and SRE interviews probe them because the fixes (window sizes, buffer tuning, a different congestion controller, AQM) are counter-intuitive, and networking research starts from the same vocabulary.

Core ideas

Two windows. The receiver advertises a receive window (rwnd): how many more bytes it can buffer. That is flow control, end to end, about the receiver's memory and reading speed. The sender also keeps a congestion window (cwnd): its estimate of how much data the path can carry without overload. At most min(rwnd, cwnd) bytes may be unacknowledged at any time.

Bandwidth-delay product. To keep a path busy, the data in flight must cover one round trip: BDP = bottleneck bandwidth x RTT. A 100 Mbit/s path with an 80 ms RTT needs 100,000,000 x 0.08 / 8 = 1,000,000 bytes, about 1 MB, in flight. If either window is smaller, throughput is capped at window / RTT however fast the link is; that is why TCP window scaling exists and why long-distance transfers need large socket buffers.

Queueing and delay. RTT = propagation delay + transmission time + queueing delay. When packets arrive faster than the bottleneck sends them, a queue forms there. A queue that absorbs a burst and drains is useful; a queue that stays full only adds delay. Oversized, permanently full buffers are called bufferbloat, and they are why a video call stutters next to a bulk upload. Jitter, the variation in delay, mostly reflects changing queue occupancy. Active queue management (CoDel, FQ-CoDel, RED) drops or marks packets early to keep queues short; with ECN, routers mark packets instead of dropping them.

Loss-based congestion control. Reno starts with slow start, doubling cwnd every round trip until a loss or a threshold, then performs additive increase (about one segment per RTT) and multiplicative decrease (halving on loss). AIMD drives flows with similar RTTs toward equal shares. After a timeout the sender restarts from a small window. CUBIC, the default in Linux, Windows and Apple stacks, grows its window as a cubic function of the time since the last loss, which scales better on high-BDP paths, but it still treats loss as the main signal and therefore tends to fill buffers.

Other signals. Delay-based algorithms (Vegas and successors) back off when the RTT rises above its minimum; BBR builds a model from bottleneck bandwidth and minimum RTT and paces at the estimated rate. Which works best depends on buffer size, competing traffic and the version deployed, which is the starting point of congestion-control research.

Common misconceptions

• "Flow control and congestion control are the same thing." Flow control protects the receiver and is signalled explicitly; congestion control protects the network and is inferred from loss, delay or ECN marks.
• "Bigger buffers always help." They absorb bursts, but a permanently full buffer only adds latency.
• "Low throughput means a slow link." With a small window, throughput = window / RTT; compute the BDP before blaming bandwidth.
• "Packet loss always means congestion." Wireless errors and traffic policers also drop packets, which is one reason loss-based control underperforms there.
• "Slow start is slow." The window grows exponentially; the name refers to starting small instead of sending a full window at once.

How to approach it

Practice the BDP calculation with units until it is automatic. Draw cwnd over time for Reno (the sawtooth) and label each phase. Connect it to observation: ss -ti shows cwnd, RTT, retransmissions and whether a flow is limited by the receiver window or the application. Mid and senior questions cover tuning long-haul transfers, why latency rises under load, what AQM changes, and CUBIC versus BBR tradeoffs without claiming either wins everywhere.