Network troubleshooting and measurement

Why it matters

Prompts such as "the service is slow from one region" or "some requests time out" appear in SRE and infrastructure interviews and in real incidents alike. Strong answers form hypotheses, pick the cheapest measurement that separates them and read the evidence carefully; weak answers run tools at random or blame "the network" without isolating which hop or phase is slow.

Core ideas

Decompose the request. A request has phases: name resolution, TCP connect (one round trip), TLS handshake, time to first byte (network plus server processing) and transfer. curl's timing variables (time_namelookup, time_connect, time_appconnect, time_starttransfer, time_total) are cumulative timestamps for these phases, so the differences between them isolate each step. Compare cold and warm runs and different client locations. High connect time points at the path or at SYN handling; high time to first byte with fast connect points at the server or what is behind it.

DNS. dig shows what a resolver returns, which server answered, the remaining TTL and the query time. dig +trace follows the referrals from the root; asking the authoritative server directly and comparing with your resolver separates stale caches from wrong records. Applications and runtimes may also cache on their own.

Socket state. ss, the successor to netstat, lists sockets with their states and queues. With ss -tan: many SYN-SENT means connects are not completing; many CLOSE-WAIT means the local application is not closing sockets; many TIME-WAIT reflects connection churn. For a listening socket, Recv-Q near the backlog signals accept-queue overflow. ss -ti adds TCP internals per connection: smoothed RTT and its variation, cwnd, retransmissions, delivery rate, and whether the flow is limited by the receiver window or by the application.

Packets. tcpdump captures with a filter (host, port, protocol) and writes pcap files for Wireshark. Read timing first: the gap between SYN and SYN-ACK is the network RTT seen by the client; the gap between the request and the first response byte adds server time; retransmissions, duplicate ACKs, zero windows and resets each point to different causes. Capture at both ends when you can: a packet that leaves one host and never arrives localizes the problem to the path.

Path tools. ping measures RTT and loss to a host, though ICMP may be deprioritized or blocked. traceroute and mtr list hops and per-hop delay, but routers often rate-limit their replies, so loss at an intermediate hop that does not continue to later hops is usually not real loss. MTU problems show up as connections that complete the handshake but stall on large transfers.

Method. Go from symptom to hypotheses to the cheapest discriminating test; change one variable at a time; write down what each test ruled out.

Common misconceptions

• "Ping works, so the network is fine." ICMP may be treated differently, and TCP-level problems (resets, MTU, overflowing accept queues) do not show up in ping.
• "Loss at one traceroute hop means that router drops traffic." Routers deprioritize replies to probes; only loss that persists to the destination matters.
• "High latency means the network is slow." Time to first byte often includes server queueing; separate connect time from server time first.
• "tcpdump shows exactly what is on the wire." With segmentation and checksum offloads, captures on the sending host can show oversized segments or bad checksums that never existed on the wire.
• "CLOSE-WAIT build-up is a network problem." The local application received the peer's FIN and never closed its socket.

How to approach it

Practice on real traffic: time requests to several sites with curl, capture a handshake, read ss output on a loaded machine. In interviews, narrate a plan before naming tools: what you expect each measurement to show and how it narrows the search. At senior level add fleet thinking: compare healthy and unhealthy hosts, correlate with deploys, and start from metrics and traces before taking packet captures.