Design a chat system
Design WhatsApp/Messenger: persistent WebSocket connections vs long-poll, a connection registry for routing, message delivery with per-conversation ordering and the three delivery ticks (sent/delivered/read), presence at scale, offline queues, group fan-out, and message storage.
A team built a chat MVP on plain HTTP polling: every client asked “any new messages?” every two seconds. It demoed fine. At a few hundred thousand users it collapsed — not from message volume, which was tiny, but from the relentless empty polls. Ninety-eight percent of requests returned “nothing new”, yet each one cost a full TCP/TLS handshake, an auth check, and a database query. The servers were pinned at 90% CPU serving nothing. The team’s first reaction was to poll less often, which made the app feel laggy without fixing the load. The real problem was architectural: chat is a system where the server must push to the client the instant a message arrives, and a request/response model fundamentally cannot do that without wasteful polling. This lesson designs the push-based system underneath WhatsApp-style chat — connections, routing, ordering, receipts, presence, and storage.
Requirements
Before listing features, ask yourself: what is fundamentally different about chat compared to a typical request/response API? The answer shapes every architectural decision — the connection tier, the registry, the ordering guarantee.
Functional: one-to-one and group messaging; messages delivered in near-real-time when the recipient is online and queued for delivery when offline. Delivery receipts — sent, delivered, read (the familiar ticks). Presence — online/last-seen/typing. Ordering — messages in a conversation appear in a consistent order for all participants. Message history is persisted and syncs across a user’s devices.
Non-functional: low latency (a message should land in well under a second when both parties are online). Reliable delivery — a sent message is never lost, even across disconnects, crashes, and offline recipients; this implies at-least-once delivery plus dedup. Huge connection count — hundreds of millions of devices each holding a long-lived connection, which is a fundamentally different scaling problem from request/response (you scale by concurrent connections, not RPS). Eventual consistency is fine for presence and read receipts; ordering within a conversation must be consistent.
The defining property: the server must initiate delivery (push), so the architecture is built around persistent connections and routing a message to whichever server currently holds the recipient’s connection — the opposite of stateless HTTP.
Estimation
500M daily active users, most keeping the app connected, means on the order of hundreds of millions of concurrent long-lived connections. A single server can hold tens to low-hundreds of thousands of idle WebSockets (memory and file descriptors per connection are the limit, not CPU), so you need thousands of connection-gateway servers just to terminate connections. Message volume: if each user sends ~40 messages/day, that is 2×10^10 messages/day, about 250,000 messages/second average, multi-x at peak. Each message is small (hundreds of bytes), so storage is dominated by count, not size: ~20 billion messages/day at a few hundred bytes is several TB/day, demanding a write-optimized, horizontally-sharded message store with retention policy.
The napkin’s verdict: the scaling axis is concurrent connections, which forces a dedicated, stateful connection tier separate from the stateless business logic — and a way to find which gateway holds a given user.
High-level design
The system splits into a stateful connection tier and a stateless routing/storage core.
- Connection gateways terminate the persistent WebSocket connections. They are stateful (they hold sockets) and exist only to receive from and push to clients.
- Chat service is stateless: it persists the message (assigning a per-conversation sequence number), looks up the recipient in the registry, and forwards for delivery.
- Connection registry maps
user/device, gatewayso any chat service instance can find where to send a message. It is updated on connect/disconnect. - Message store is the durable history, sharded by conversation, ordered by sequence number.
- Offline queue holds messages for disconnected users until they reconnect and sync.
Deep dive
WebSockets vs long-poll, and the connection tier
HTTP is request/response: the client must ask before the server can answer. For chat that means polling, and the hook shows why polling fails — empty polls dominate and each carries full request overhead. The fixes, in order of quality:
- Long-polling: the client makes a request that the server holds open until there is a message (or a timeout), then responds; the client immediately re-requests. This eliminates empty-poll spam but still pays a new request per message and is awkward for server-push bursts. It is the fallback for environments without WebSockets.
- WebSocket: a single TCP connection, upgraded from HTTP once, that stays open for bidirectional messaging with tiny per-message framing overhead. The server can push the instant a message arrives. This is the primary transport for modern chat.
Because connections are long-lived and stateful, the connection tier is its own thing: you scale it by adding gateways, and a gateway crashing drops its connections (clients reconnect, ideally to a different gateway, with backoff and jitter to avoid a reconnect thundering herd). The registry must update on every connect/disconnect, which is why it is a fast, in-memory store (Redis-style) and not a relational table — connection churn is constant.
▸Why this works
Why separate the stateful connection gateways from the stateless chat logic instead of doing everything in one process? Because the two have opposite scaling and failure properties. The connection tier scales by concurrent connections and is memory/fd-bound; it holds long-lived state (the open sockets) and must survive deploys gracefully (you do not want a routine deploy to drop 100k sockets). The chat logic scales by message throughput, is CPU-bound, is stateless, and you redeploy it freely. Fusing them means every business-logic deploy disconnects everyone, and you cannot scale the two axes independently. The registry is the seam that lets them be separate: gateways own sockets and register them, the stateless service looks up the registry to route — so you can deploy logic without touching connections, and add connection capacity without touching logic.
Message delivery, ordering, and the three ticks
Reliable delivery is at-least-once with dedup. The sender’s client assigns a client-side message id; the server persists the message before acking the sender (so a crash after ack never loses it), and the recipient dedups on the message id (so a re-delivery after a reconnect does not duplicate). This gives the user effectively-once.
Ordering is per-conversation, not global — there is no meaningful global order across all of WhatsApp, only “within this chat, in what order did messages happen”. The clean way is to assign a monotonic sequence number per conversation when the message is persisted, and have clients render by sequence. Because all messages of one conversation are routed through and stored on the same shard (sharded by conversation id), a single sequence authority per conversation gives a total order without global coordination. Clients reconcile out-of-order arrivals (network reordering, retries) using the sequence number.
The three ticks are three delivery states that flow back to the sender as separate events:
- Sent — the server has persisted the message (one tick).
- Delivered — the recipient’s device received it, i.e. it was pushed down their socket and acked by the client (two ticks).
- Read — the recipient opened the conversation (two blue ticks).
Each is an acknowledgment traveling the reverse path. Delivered and read receipts are themselves small messages routed back to the sender (and can be batched/eventual — nobody needs a read receipt in 10 ms).
▸Common mistake
A classic correctness bug is acking the sender before the message is durably stored — “optimistically” replying “sent” the moment the gateway receives it, then persisting asynchronously. It feels faster, but if the server crashes between the ack and the write, the message is gone while the sender’s UI shows a confident single tick. The message is silently lost, which is the one thing a chat system must never do. The rule: persist before you ack. The sender’s “sent” tick must mean “the server has it durably”, not “the server saw it”. The same discipline applies to the recipient side: the “delivered” receipt must mean the client durably received and acked it, so that a recipient crash mid-delivery re-delivers on reconnect (dedup makes that safe) rather than dropping the message. Ordering bugs have the same root — assign the sequence number at persist time, the single point where the message becomes real, not at receive time on a gateway that might be one of several racing.
Presence, offline delivery, and group chat
Presence (online / last-seen / typing) is high-churn, low-value-per-update state, and treating it like messages would swamp the system. It lives in a fast in-memory store keyed by user, updated by gateway connect/disconnect and heartbeats, and read on demand. The scaling trap is presence fan-out: naively pushing “X came online” to all of X’s contacts means a popular user’s connect event fans out to thousands of pushes. So presence is usually pulled (a client fetches the presence of the contacts currently visible on screen) or pushed only to a bounded interested set, and it is explicitly eventual — a slightly stale “last seen” is acceptable, an undelivered message is not.
Offline delivery: if the registry shows the recipient has no live connection, the message is persisted (it always is) and flagged for the offline queue. On reconnect, the client syncs everything since its last-acked sequence number per conversation — the sequence number doubles as the sync cursor. This is why persistence is unconditional: online delivery is an optimization on top of a store-and-sync foundation, not a replacement for it.
Group chat is fan-out again. A message to a 500-member group must reach up to 500 recipients. Small groups fan out on write (push to each member’s delivery path); very large groups/broadcast channels lean toward fan-out-on-read or a hybrid, exactly like the news feed — the same power-law logic. Each member’s delivery still flows through the registry to their gateway (or to their offline queue), and each maintains their own read position in the conversation.
Your chat backend is pinned at 90% CPU with a few hundred thousand users, but message volume is tiny and 98% of requests return 'nothing new'. What is the architectural fix?
Two users on different gateways message rapidly; messages sometimes render out of order, and after a reconnect one message is duplicated. How do you get consistent ordering and no duplicates?
To route a message to an online recipient, a stateless chat service looks the recipient up in the _______ — the fast, in-memory map from each connected user/device to the connection gateway currently holding their socket — then forwards the message to that gateway, which pushes it down the socket.
You are designing the real-time delivery transport for a chat system serving 500M daily active users. Most clients stay connected for hours and expect sub-second message delivery. Which transport model should back the connection tier?
Bottlenecks & tradeoffs
The binding constraint is concurrent connections, not request rate — chat scales on a different axis from typical web services, which is why it needs a dedicated stateful gateway tier. Key tradeoffs:
- WebSocket vs long-poll vs polling. WebSocket gives true bidirectional push at minimal per-message cost but needs sticky, stateful connections and graceful reconnect handling; long-poll is the compatibility fallback; plain polling is wasteful and wrong for push (the hook).
- Per-conversation ordering vs global ordering. Per-conversation sequence gives a total order where it matters with no global coordination; insisting on a global order buys nothing and bottlenecks everything.
- At-least-once + dedup vs exactly-once. Reliable delivery across reconnects means at-least-once plus a message id for dedup (persist before ack); true exactly-once over an unreliable network is unattainable.
- Presence: push vs pull, and eventual. Presence is high-churn and low-stakes, so it is eventual and usually pulled for visible contacts; pushing every presence change fans out catastrophically for popular users.
- Group fan-out: write vs read. Small groups fan out on write; huge groups/channels use read-side or hybrid fan-out — the same power-law tradeoff as the feed.
The deepest principle: persistence is unconditional and online delivery is an optimization on top of store-and-sync. Persist before you ack; use the per-conversation sequence number as both the order and the sync cursor; and never let the “fast path” (push) become a way to lose a message the “slow path” (store, then sync on reconnect) would have kept.
- 01Why does chat need WebSockets and a separate connection tier rather than HTTP polling?
- 02How are reliable delivery and consistent ordering achieved, and what is the persist-before-ack rule?
- 03What are the three ticks and how does each travel?
- 04How are presence and offline delivery handled at scale, and why is persistence unconditional?
A chat system is fundamentally server-push, which is why polling fails (the hook: empty polls pin the CPU while delivering nothing) and why the architecture centers on persistent WebSocket connections. Those connections are long-lived and stateful, so they live in a dedicated connection-gateway tier scaled by concurrent connections, separate from the stateless, CPU-bound chat logic; the connection registry (a fast in-memory map from user/device to gateway) is the seam that lets a stateless service route a message to the one server that can push it, and lets the two tiers scale and deploy independently. Delivery is at-least-once with dedup on a client message id, and the cardinal rule is persist before you ack — the “sent” tick must mean durably stored. Ordering is per-conversation via a monotonic sequence number assigned at persist time on the conversation’s shard, which also doubles as the offline sync cursor. The three ticks (sent/delivered/read) are acknowledgments flowing back to the sender. Presence is eventual, high-churn, and pulled rather than fanned out; offline delivery rests on unconditional persistence plus reconnect-and-sync; and group chat fans out write-side for small groups and read-side/hybrid for huge ones — the same power-law logic as the news feed. The unifying idea: online push is an optimization layered over a durable store-and-sync core, never a substitute for it. Now when you see messages arriving out of order or users complaining about duplicates after a reconnect, you know exactly where to look: the per-conversation sequence number, the persist-before-ack rule, and the client-side message id for dedup.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.