WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10M

Phoenix demonstrated 2 million concurrent WebSocket connections on one 64 GB server. MigratoryData reports 10 million on commodity hardware. Getting there requires understanding three independent ceilings: RAM per idle connection, file descriptor limits, and NIC bandwidth — and the protocol evolution that shifts those ceilings.

Three ceilings in order

Ceiling 1 — RAM per idle connection. An idle WebSocket connection consumes kernel socket buffers (send + receive), plus the framework’s application state. At default Linux settings:

• Kernel socket buffers: ~4–8 KB (socket rmem/wmem default).
• Framework connection state: varies by language/framework (Node.js: ~10–50 KB per connection due to V8 object overhead; Go: ~2–5 KB; Rust: ~1–2 KB).

With permessage-deflate at default window_bits=15: +256 KB per compressor + 44 KB per decompressor = +300 KB per connection.

At 10 M connections × 10 KB base = 100 GB RAM minimum. This is why Phoenix’s 2 M on 64 GB is impressive — they strip per-connection overhead to under 32 KB per connection.

Ceiling 2 — File descriptors. Linux defaults are 1,024 per process. Production servers raise with ulimit -n or LimitNOFILE in systemd. Maximum practical limit per process is around 1 M on modern Linux. At 10 M connections per server you need 10 M file descriptors — requires kernel tuning (fs.nr_open, fs.file-max).

Ceiling 3 — NIC bandwidth. Keepalive pings every 25 s at 125-byte payload: 10 M connections × 4 pings/minute × 125 bytes = ~83 MB/s of ping traffic alone on a 1 Gbps NIC. NIC saturation sets the practical ceiling for idle connections around 500 k–2 M depending on ping interval and message rate.

HTTP/2 extended CONNECT (RFC 8441)

Opening 100 WebSocket connections via HTTP/1.1 Upgrade means 100 TCP 3-way handshakes (~30 ms each at 10 ms RTT) plus 100 TLS sessions (~20 ms resuming). Each TCP connection consumes a file descriptor and ~8 KB of kernel buffers.

RFC 8441 defines extended CONNECT for WebSocket over HTTP/2. Instead of GET + Upgrade, the client sends:

HEADERS stream=5
  :method = CONNECT
  :protocol = websocket
  :scheme = https
  :path = /chat
  :authority = example.com
  sec-websocket-protocol = chat

The server replies with 200 OK (not 101). The HTTP/2 stream 5 becomes the WebSocket tunnel. Other HTTP/2 streams on the same TCP connection carry different traffic simultaneously.

Benefits:

• 100 WebSocket connections share one TCP connection — one 3-way handshake total.
• Zero extra file descriptors for additional connections.
• Congestion control tuning on the shared connection benefits all streams.
• Cumulative handshake latency for 100 connections: ~30 ms instead of ~3,000 ms.

Trade-off: requires HTTP/2 end-to-end (client, server, all proxies). As of 2026, major cloud load balancers (AWS ALB, GCP Load Balancer) support it, but most corporate proxies and firewalls do not. Adoption is limited to hyperscale operators.

WebSocket over HTTP/3 (RFC 9220)

HTTP/3 runs over QUIC (UDP-based). RFC 9220 defines WebSocket over HTTP/3 using the same extended CONNECT with :protocol websocket.

Key difference from HTTP/2: QUIC streams are independent at the transport layer — a lost packet on stream 0 does not block stream 4 (no head-of-line blocking). HTTP/2 over TCP still suffers TCP-level HoL blocking: a lost TCP segment blocks all HTTP/2 streams on that connection.

As of early 2026, no major browser or server ships production WebSocket-over-HTTP/3 support. WebTransport (a QUIC-native protocol) is the preferred choice for applications needing QUIC semantics: it offers bidirectional streams plus unreliable datagrams and has ~75% browser support (Chrome 120+, Firefox 127+, Safari 17+).

permessage-deflate tuning

The permessage-deflate extension (RFC 7692) compresses each message with DEFLATE. Compression ratio:

• 50–90% on text payloads (JSON, HTML).
• Poor on small messages (< 64 bytes) — can expand.
• ~0% on already-compressed data (images, video, encrypted payloads).

Memory cost per connection at default window_bits=15:

• Compressor: ~256 KB
• Decompressor: ~44 KB

At window_bits=12: ~50 KB total per connection. The trade-off is lower compression ratio (32 KB history window vs. 32 KB at window_bits=15… wait, window_bits=15 = 32 KB, window_bits=12 = 4 KB). Compression ratio drops from ~70% to ~50% on typical JSON payloads.

Production strategies:

• Disable entirely for idle connections and connections sending small messages or binary data.
• Set window_bits=10–12 for memory-constrained deployments.
• Enable context_takeover=false to reset compression state per message — trades ratio for lower per-message overhead.