WebSocket: survive the broadcast storm

Build a WebSocket broadcast service (chat or live-feed), deliberately drive it into a backpressure OOM and a reconnect thundering herd, then harden it with a per-connection high-water mark, jittered exponential backoff, and a Redis pub/sub fan-out — proving each fix with before/after measurements under identical load.

• Build a WebSocket broadcast server in any stack (Go, Node, Rust, Elixir) that accepts upgrades, validates the Origin header, sends a server-side ping every 25-30 s, and broadcasts a configurable payload to all connected clients on an interval.
• Write a load client that opens N connections and can simulate slow readers (a fraction of clients that read far slower than the broadcast rate, e.g. by throttling their receive loop).
• Wire observability: active connection count, per-connection send-queue depth (p95/p99), total queued bytes, slow-client count, and close-code distribution (1000 / 1006 / 1013) — exported to logs or a metrics endpoint.
• Reproduce the backpressure failure: broadcast a large payload on a tight interval to thousands of clients with a few percent slow readers; capture the baseline showing queued bytes and process RSS climbing toward OOM.
• Apply the high-water mark: cap each connection's send queue (e.g. 100 messages or 10 MB) and on breach either evict with close 1013 or drop stale messages; show queued bytes and RSS now hold flat under the same load.
• Reproduce the reconnect thundering herd: kill the server while M clients are connected, with the client using fixed-interval (or jitter-free) backoff; capture the synchronized SYN waves and the failed/slow recovery.
• Apply jittered exponential backoff (random base 100-500 ms, double per attempt, cap 30-60 s, plus a random per-delay jitter factor) and show the reconnects now spread across a window and the server recovers in one or two retry windows.
• Add a second server behind a load balancer with sticky sessions and a Redis Streams (or equivalent) pub/sub backbone; show that a message sent to a client on server A reaches a client on server B.

• A before/after table for backpressure: total queued bytes, process RSS, slow-client count, and p99 broadcast latency under identical load — measured, not estimated — showing RSS flat after the high-water mark instead of climbing to OOM.
• A before/after comparison for reconnection: SYN/accept rate over time and time-to-full-recovery, showing the jittered version spreading retries across a window and recovering, versus the synchronized version stalling.
• A demonstration that the Origin check rejects an unauthorized origin (403) and that an oversized message is rejected with close code 1009.
• A short write-up naming, for each fix, which lever from the unit you used (high-water mark, jitter, pub/sub fan-out, sticky sessions) and why it was the highest-leverage choice.