SYN cookies, TFO, and TIME-WAIT at scale

A busy load balancer starts returning EADDRNOTAVAIL on outbound connections to a backend. ss -tan shows 30,000 TIME-WAIT sockets. Simultaneously, the public listener is being probed with a SYN flood. Both problems have solutions in Linux sysctls — but the wrong sysctl makes things worse.

SYN cookies internals

Under a SYN flood, the server’s half-open queue overflows. Instead of allocating a request_sock for every SYN, the kernel takes the path in net/ipv4/tcp_ipv4.c:cookie_v4_init_sequence:

ISN_in_SYN-ACK = MAC over (saddr, sport, daddr, dport, isn, secret)

The top 5 bits encode the MSS index (8 options), the next 6 bits are a minute counter (replay window), and the low 21 bits are an HMAC. The connection is forgotten immediately. When the ACK comes back, cookie_v4_check recomputes and validates. If valid, the kernel allocates a full socket. If invalid, the ACK is silently dropped.

The trade-off: CPU for memory — fine for large servers, not for small ones. The cost: only MSS survives in the cookie. Window scaling, SACK, and Timestamps offered in the original SYN are silently dropped for cookie-validated connections. On a long-RTT high-bandwidth path during a flood, legitimate clients’ throughput degrades — the window is capped at 64 KiB without scaling.

Linux enables SYN cookies when the half-open queue exceeds tcp_max_syn_backlog (default system-dependent, typically 4096). Enable with net.ipv4.tcp_syncookies=1 (default since kernel 2.4). Value 2 forces unconditional cookie generation for testing.

$ sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0' -tnn
14:23:45.123456 IP 203.0.113.50.34521 > 198.51.100.10.443: Flags [S], seq 1000000, win 29200, length 0
14:23:45.123489 IP 198.51.100.10.443 > 203.0.113.50.34521: Flags [S.], seq 3851629874, ack 1000001, win 28960, length 0
14:23:45.124000 IP 203.0.113.50.34522 > 198.51.100.10.443: Flags [S], seq 2000000, win 29200, length 0
14:23:45.124050 IP 198.51.100.10.443 > 203.0.113.50.34522: Flags [S.], seq 4025814937, ack 2000001, win 28960, length 0
14:23:45.125000 IP 203.0.113.50.34521 > 198.51.100.10.443: Flags [.], seq 1000001, ack 3851629875, win 29200, length 0
14:23:45.125010 IP 203.0.113.50.34522 > 198.51.100.10.443: Flags [.], seq 2000001, ack 4025814938, win 29200, length 0
14:23:45.200000 IP 203.0.113.100.55000 > 198.51.100.10.443: Flags [S], seq 5000000, win 29200, length 0
14:23:45.200050 IP 198.51.100.10.443 > 203.0.113.100.55000: Flags [S.], seq 2941837465, ack 5000001, win 28960, length 0
(no ACK from 203.0.113.100; SYN flood from different spoofed sources)

TCP Fast Open (TFO, RFC 7413)

The standard handshake costs 1 RTT before application data. TFO eliminates that for subsequent connections:

1. First connection: server sends a Fast Open Cookie option in SYN-ACK; client caches per-(server-IP, server-port).
2. Subsequent connections: client puts the cookie in SYN along with up to MSS bytes of application data; server validates and processes the data while completing the handshake — saving 1 RTT.

On 280 ms RTT (London → Sydney), TFO saves 280 ms per warm connection.

Deployment reality: middleware problems killed broad server-side TFO adoption. Middleboxes (firewalls, DPI appliances, some NAT gateways) strip the TFO option from SYN, breaking cookie delivery. Some NAT gateways break cookies when the client’s external IP changes. Fallback is mandatory but adds latency. Server deployment is limited; most 0-RTT interest has shifted to QUIC, which sits above UDP and bypasses protocol ossification in network infrastructure. Linux supports both sides since kernel 3.6 (2012) via net.ipv4.tcp_fastopen bitmask (1=client, 2=server, 3=both).