TCP options and common pathologies

A Redis client call takes 200 ms even though Redis is on the same datacenter host. No load, no timeouts, no errors — just a 200 ms delay on every small request. The cause is two TCP defaults interacting in the worst possible way, and fixing it takes one socket option.

TCP header anatomy

Every TCP segment carries a 20-byte mandatory header plus optional fields (up to 40 bytes more):

The options field (up to 40 bytes) carries MSS, Window Scale, SACK, Timestamps, and TFO cookie. Options are negotiated only in SYN/SYN-ACK — once the connection is established, the option set is fixed.

TCP options field

Beyond MSS, Window Scale, and SACK:

Timestamps (RFC 7323): used for both RTT measurement and PAWS (Protection Against Wrapped Sequence numbers). PAWS prevents old delayed packets from being mistaken for new data on long-lived high-bandwidth connections where sequence numbers wrap around.

The TFO cookie also lives in the options field. See the SYN cookies and TFO lesson for details.

Nagle’s algorithm and delayed ACK — the 200 ms trap

By default a TCP sender batches small writes via Nagle’s algorithm: do not send a partial segment if there is unacknowledged data in flight. Independently, the receiver delays ACKs up to 40 ms (Linux default) hoping to piggyback the ACK on outgoing application data.

Combined, these two create a deadly interaction on small request-response patterns:

1. Client sends a small request. The request is less than MSS.
2. Server’s TCP stack holds the ACK up to 40 ms hoping to piggyback on an outgoing response.
3. Client cannot send its next small write because Nagle is holding it (waiting for the ACK of the first write).
4. Server’s application eventually responds and the delayed ACK fires alongside the response.

Result: a 40–200 ms stall on every small interactive exchange.

The fix: TCP_NODELAY=1 on the socket disables Nagle. gRPC, Redis clients, HTTP/2 implementations, and any latency-sensitive RPC layer set it by default. TCP_QUICKACK=1 tells Linux to ACK the next segment immediately (it auto-resets after one packet, so call it after each read() in a tight loop).

PSH flag

The PSH (push) flag tells the receiver’s TCP stack to deliver buffered data to the application immediately rather than waiting for more. Modern stacks pass data up to the application as soon as it arrives, so PSH is more a hint than a mandate. It is set on the last segment of every write() that completes a logical message — useful when the kernel’s TCP stack coalesces multiple application writes into one segment, ensuring the receiver still delivers the record.

Explicit Congestion Notification (ECN)

Instead of dropping packets to signal congestion, ECN-capable routers mark them with a 2-bit code (CE = Congestion Experienced). The TCP header reserves CWR (Congestion Window Reduced) and ECE (ECN-Echo) flags. ECN negotiation happens in the SYN/SYN-ACK exchange: both sides advertise capability via ECE+CWR flags. When CE-marked packets arrive, the receiver sets ECE to inform the sender; the sender reduces cwnd and sets CWR to confirm.

ECN is enabled by default in Linux and macOS for connections to known-good destinations. Some old middleboxes drop ECN-marked packets — deployments use fallback detection and disable ECN to those destinations.

Keepalive

By default a TCP connection sends no packets when idle, so a connection through a NAT or firewall may be silently dropped after 5–60 minutes. SO_KEEPALIVE sends a probe every tcp_keepalive_time seconds (Linux default: 7200 s = 2h — far too long for service-mesh use). For long-lived RPC connections, tune:

• tcp_keepalive_time: 60–120 s
• tcp_keepalive_intvl: 10–30 s
• tcp_keepalive_probes: 3–5

This detects dead peers within a minute rather than two hours.

$ ss -tan state established | wc -l
12384
$ ss -tan state close-wait | wc -l
9821
$ ss -tan state time-wait | wc -l
1247
$ ss -s
Total: 12500
TCP:   23552 (estab 12384, closed 8920, orphaned 2, timewait 1247)
$ ps -p 1234 -o pid,stat,rss,vsz,cmd
PID STAT  RSS    VSZ CMD
1234 Ssl 8392000 12000000 /usr/bin/app-server