Reconnection: jittered backoff, thundering herd, message resumption

Your server restarts for a deploy. All 10 million connected clients instantly get close code 1006. They all retry at exactly 1 second. Ten million SYN packets hit your load balancer simultaneously. It collapses. The server never recovers. WebSocket has no built-in reconnection — which means you are one naive retry loop away from killing your own service during a maintenance window.

What happens on disconnect

When a WebSocket connection drops — network glitch, server restart, proxy timeout — the client receives a close event with close code 1006 (abnormal closure — no close frame was received). The application must decide: reconnect, and if so, when and how.

WebSocket itself provides zero reconnection logic. The application owns this entirely.

The thundering herd problem

If every client retries at the same fixed interval after a disconnect, the retries arrive at the server in synchronized waves:

• T=0: service goes down, 10 million clients disconnect.
• T=1s: all 10 million clients retry simultaneously.
• Server gets 10 million SYN packets in ~1 second.
• TCP SYN backlog fills. New connections are dropped.
• T=2s: all 10 million retry again at the same interval.
• Server never gets a quiet moment to recover.

Even exponential backoff without jitter fails: the synchronized doubling still creates synchronized bursts.

Jittered exponential backoff

The fix is to add randomness (jitter) to each client’s retry timing so the 10 million retries spread over a window of seconds instead of arriving simultaneously.

Algorithm:

attempt = 0
base_ms = random(100, 500)      // different per client
max_ms  = 60_000

while not connected:
  delay = min(base_ms * 2^attempt, max_ms)
  delay = delay * random(0.5, 1.5)   // ±50% jitter
  sleep(delay)
  attempt += 1
  try_connect()

Example spread for a 100k-client server restart:

• Client A picks base=150 ms, jitter → retries at 186 ms
• Client B picks base=340 ms, jitter → retries at 412 ms
• Client C picks base=210 ms, jitter → retries at 288 ms

The 100k retries are spread over 10+ seconds. The server processes them in small batches and recovers within 1–2 retry windows instead of being crushed.

Message resumption and at-least-once delivery

Reconnection re-establishes the connection. It does not restore lost messages. Any messages the server sent while the client was disconnected — or that the client sent but the server had not yet ACK’d — are gone unless the application tracks them.

The pattern:

1. Every message gets a client-generated ID. The sender keeps the message in a retry queue until it receives an ACK for that ID.
2. The server ACKs each message ID. The sender removes ACK’d messages from its retry queue.
3. On reconnect, the client resends unACK’d messages. The server checks its Redis stream or database: if the message ID already exists, it sends ACK to the client but does not republish (prevents duplicates for other subscribers).

This implements at-least-once delivery with idempotent deduplication — no message is lost, no message is delivered more than once to other clients.

For the server to deduplicate, it needs durable state that survives restarts. The typical store:

• Redis stream (fast, persistent, ordered, with configurable retention).
• PostgreSQL (durable, queryable, suited for message history with audit requirements).

The client reconnect flow:

1. Reconnect with backoff.
2. On connection established, send { type: "resume", lastSeenId: "msg-4291" }.
3. Server reads the stream from msg-4292 onward and delivers missed messages.
4. Server ACKs the delivery.
5. Client confirms receipt and updates its lastSeenId.