TCP Beyond the Basics

Learning goals

Go beyond the three-way handshake from chapter 2: explain flow control vs congestion control, why head-of-line blocking hurts HTTP/1.1, what TIME_WAIT means in ops, and when Nagle’s algorithm still matters—enough to debug latency and connection churn without becoming a kernel developer.

Assumed context

You finished the HTTPS chapter: TCP establishes a connection, TLS runs on it, HTTP exchanges messages. Here we zoom into TCP behavior that explains slow transfers, stuck connections, and why HTTP/2 and QUIC exist.

Plain English: TCP tries to deliver every byte in order and without loss. That reliability has costs—buffers, retransmits, and global congestion rules every host on the internet shares.

Flow control (receiver’s pace)

Each side advertises a receive window: “I have this much buffer space left.”

If the sender outruns the receiver, the window shrinks to zero and the sender pauses. This is end-to-end between two hosts—not about internet-wide congestion.

Example: a busy application reads the socket slowly; the kernel receive buffer fills; TCP tells the peer to stop sending until the app catches up. Symptom: low throughput on one connection while CPU or thread pool is saturated locally.

Precise term: Flow control uses the window field in TCP headers (and window scaling in modern stacks) so a fast sender cannot overrun a slow receiver.

Congestion control (network’s pace)

Separate from flow control: TCP guesses how much traffic the path can handle. Classic algorithms (slow start, congestion avoidance, fast retransmit/recovery) adjust a congestion window (cwnd) based on loss and delay signals.

Plain English: after handshake, TCP does not flood the link immediately—it probes capacity. Packet loss often triggers backoff (throughput drops). High latency on a clean link may mean buffers bloating along the route, not your app alone.

Why you care:

• Long fat networks (high bandwidth × high RTT) need time to ramp up—short requests may never reach full speed.
• Many parallel TCP connections (browser, microservices) compete for the same bottleneck.

Head-of-line (HOL) blocking

TCP delivers an ordered byte stream. If segment 2 is lost, segment 3 waits in the receiver—even if segment 3 belonged to a different HTTP response on the same connection.

HTTP/1.1 pipelining tried multiple requests on one connection but was largely abandoned due to HOL at the HTTP layer. HTTP/2 multiplexes many streams on one TCP connection—better, but TCP-level HOL remains: one lost packet stalls all streams on that connection.

That limitation motivates QUIC over UDP (next chapter): independent streams at transport layer.

TIME_WAIT

When a side actively closes a TCP connection, it enters TIME_WAIT (often ~60 s on Linux) after sending/receiving the final ACK. Purpose: handle delayed duplicates so a new connection with the same 4-tuple does not get confused with the old one.

Ops symptoms:

• Cannot assign requested address or ephemeral port exhaustion under heavy short-lived client traffic.
• Many TIME_WAIT sockets in ss -tan state time-wait after load tests.

Mitigations (use with care): connection pooling/reuse, tune ip_local_port_range, tcp_tw_reuse where appropriate, fix unnecessary connect-per-request patterns. Do not disable TIME_WAIT blindly—it protects correctness.

Plain English: closing lots of TCP connections quickly leaves sockets in a waiting room; high-QPS clients should reuse connections (HTTP keep-alive, pool).

Nagle’s algorithm (brief)

Nagle coalesces small sends: if there is unacknowledged data, buffer tiny writes until an ACK arrives or enough data accumulates. Reduces tiny packets (“packet overhead”) but can add latency for request/response patterns with small messages.

Most stacks enable Nagle by default. TCP_NODELAY disables it—common for interactive or low-latency RPC when you accept more small packets.

Rule of thumb: if you see 200 ms delays on small writes with no other explanation, check Nagle vs delayed ACK interaction. Many HTTP servers disable Nagle for established connections.

Tuning intuition (not a checklist)

Always measure (RTT, retransmits, curl -w, tracing) before sysctl shopping.

TCP and TLS together

TLS records sit atop the byte stream. Loss and reordering affect TLS the same as HTTP—retransmits delay handshake and application data alike. 0-RTT and session tickets (covered in TLS chapters) reduce repeated handshake cost when you must open new TCP connections frequently.

Common mistakes (beginners)

1. Confusing flow control with congestion control. — Receiver buffer vs network capacity; different mechanisms.
2. Opening a new TCP+TLS connection per API call. — Handshake RTT dominates; reuse with keep-alive or pools.
3. Blaming “TCP is slow” for application blocking. — Thread pools, DNS, DB queries often matter more—verify with traces.
4. Disabling TIME_WAIT on servers without understanding client vs server role. — Client-side ephemeral exhaustion is the usual pain.
5. Assuming HTTP/2 eliminated all HOL blocking. — Only HTTP-layer; TCP loss still blocks all multiplexed streams.

Practice checkpoint

1. Flow control vs congestion control—in one sentence each?

   • Answer: Flow control prevents overrunning the receiver’s buffer; congestion control prevents overrunning the network path.

2. Why does one lost packet on HTTP/2 still hurt all parallel requests on that connection?

   • Answer: TCP must deliver bytes in order; loss delays delivery for the entire connection, stalling all HTTP/2 streams.

3. What is TIME_WAIT protecting against?

   • Answer: Late duplicate segments from an old connection being mistaken for a new connection with the same addresses/ports.

4. When might you set TCP_NODELAY?

   • Answer: When small frequent writes need low latency and coalescing (Nagle) causes noticeable delay.

Interview angles

• “Why HTTP/3 over QUIC instead of HTTP/2 over TCP?” — Per-stream multiplexing without TCP-level HOL on loss; faster handshakes with integrated TLS (details in next chapter).
• “Explain TCP three-way handshake” — SYN, SYN-ACK, ACK; sets ISN, window scaling, options—assume known from chapter 2, deepen with cwnd starting after.
• “How would you debug high latency to an API?” — DNS, TCP/TLS handshake breakdown, TTFB vs download, retransmits, connection reuse, geographic RTT.

Next: /networking/learn/transport/udp-quic-http3/