Design a notification system

Requirements

Pin down the scope before drawing boxes — the requirements decide the architecture.

Functional: accept an event (“user X was mentioned”), resolve it to recipients, render per-channel content, and deliver across push (mobile + web), SMS, and email. Honour per-user preferences (which channels, which categories, quiet hours). Support transactional sends (password reset — must arrive, low volume) and bulk sends (marketing campaign — high volume, best-effort). Expose delivery status to product surfaces and analytics.

Non-functional: transactional notifications target seconds of end-to-end latency; bulk can take minutes. At-least-once delivery with dedup — losing a password reset is unacceptable, but a duplicate is also unacceptable to the user, so we need both a retry path and an idempotency guard. The system must absorb spikes (a viral post mentioning a celebrity fans out to millions) without dropping events or melting the providers. And it must be observable: for any send, answer “what happened and when”.

The single hardest constraint hiding in that list: third-party providers (APNs, FCM, an SMS aggregator, an email provider) are unreliable, rate-limited, and outside your control. The design is mostly about insulating yourself from them.

Estimation

Sizing tells us where the pressure is. Take 100M users, each receiving ~10 notifications/day across channels: 10^9 notifications/day. Divide by ~10^5 seconds/day, giving ~10,000 notifications/second average, and traffic is bursty (a product launch or a celebrity event spikes it), so provision for a ~5x peak, roughly 50,000/sec.

Split by channel: push dominates (~70%), email ~20%, SMS ~10% (SMS costs real money per message, so it is gated). A celebrity with 50M followers posting once is a single event that fans out to 50M sends — that one event, if processed naively in the request path, is a self-inflicted outage. Storage: delivery records at ~200 bytes each, times 10^9/day, times a 30-day retention window, lands around 6 TB of hot status data, which forces a time-series or wide-column store, not a single relational table.

The takeaway from the napkin: the fan-out amplification (one event into millions of sends) and the per-second peak both say the send must be asynchronous and queue-buffered — never synchronous in the API that accepted the event.

High-level design

The pipeline has five stages, each a buffering seam against the next stage’s failure.

1. Event API validates and persists the event, then enqueues it and returns immediately. The caller never waits for delivery.
2. Event queue decouples producers (any service that emits events) from the notification pipeline.
3. Fan-out worker resolves the event to recipients and explodes it into per-recipient work. This is where the celebrity problem lives.
4. Preference + dedup filter drops channels the user disabled, respects quiet hours, and stamps each send with an idempotency key so duplicates collapse.
5. Per-channel queues + workers each own one provider relationship, with their own rate limits, retry policy, and DLQ. A push outage backs up the push queue without touching email.

Together these five stages mean every slow or flaky provider is contained to its own lane — without step 4, the idempotency guard, the entire retry story unravels into a duplicate storm the moment a provider hiccups.

Deep dive

Fan-out and the celebrity problem

Fan-out is the step that turns one event into N sends, and N can be 50M. Doing it inline in the event API would hold a request open for minutes and exhaust connections — so fan-out is its own asynchronous stage. The worker reads the event, looks up the recipient set (followers, a topic’s subscribers, an explicit list), and writes one message per recipient into the channel queues. For huge recipient sets it must paginate and checkpoint: process the follower list in batches of, say, 10,000, committing progress so a worker crash resumes instead of restarting (and re-sending the first million).

There are two fan-out shapes, mirroring the news-feed lesson. Fan-out on write (push the send eagerly to every recipient now) is right for notifications because the action is the delivery — there is no “read later”. But for a celebrity event you blend in topic-based delivery: rather than materialize 50M individual rows, push to FCM/APNs topics (a single publish that the provider fans out to all devices subscribed to that topic), reserving per-recipient sends for personalized or preference-sensitive notifications. The hybrid keeps the common case cheap and the personalized case correct.

Provider integration: APNs, FCM, SMS, email

Each channel wraps an external provider with very different semantics, and the channel worker’s job is to normalize them.

• Push. Apple’s APNs and Google’s FCM are token-based: the device registers and hands your backend a device token that you store and must keep fresh. Tokens expire or become invalid (app uninstalled, token rotated); the provider tells you via the response, and you must prune dead tokens or you will waste sends and trip abuse limits. APNs uses long-lived HTTP/2 connections — you reuse them, you do not reconnect per send. FCM offers topics for broadcast.
• SMS goes through an aggregator (Twilio, Sinch, and similar), costs cents per message, is rate-limited per sender number, and delivery confirmation arrives asynchronously via webhook (“delivered”, “failed”, “undelivered”) — so the worker cannot know the outcome at send time; it records “submitted” and updates on the receipt.
• Email goes through an SMTP relay or API provider; bounces and complaints also arrive via webhook, and you must honour them (a hard bounce means stop sending to that address, or your sender reputation tanks and everything lands in spam).

The unifying design: a channel adapter interface (send, parse-response, classify-error) so the worker logic — rate limit, retry, record — is identical across channels, and only the adapter knows the provider’s quirks. Errors classify into retryable (timeout, 429, 5xx) and permanent (invalid token, hard bounce, unsubscribed) — permanent errors must not be retried; they go straight to cleanup.

Rate limiting, dedup, retries, and the DLQ

Four pieces of machinery keep the pipeline honest under stress.

Rate limiting is per-provider and per-recipient. Per-provider: respect APNs/FCM/aggregator quotas (a token bucket on each channel worker) so you do not get throttled or banned — the hook’s app got its APNs connection throttled for hammering it. Per-recipient: cap how many notifications a single user gets in a window (no “you have 200 new likes” as 200 separate pushes — collapse them).

Dedup / idempotency. Every send carries an idempotency key (event id + recipient id + channel). Before sending, the worker checks a fast store (Redis with a TTL) for that key; if present, skip. This is what would have saved the hook: even with the retry storm, each unique send fires once. At-least-once delivery plus an idempotency guard gives you effectively-once as the user experiences it.

Retries with backoff and jitter. Retryable failures retry with exponential backoff and jitter — not a tight loop (which becomes a DDoS on a recovering provider) and not fixed delays (which synchronize all clients into a thundering herd). Cap the attempts (say 5).

Dead-letter queue. When retries exhaust, or a message is malformed (poison message), it goes to a DLQ instead of being retried forever or dropped silently. The DLQ is inspected by on-call, alerted on, and is the difference between “we lost notifications and do not know it” and “47 sends failed permanently, here they are”. The hook had no DLQ — failures looped.

Delivery tracking

“Enqueued” is not “delivered”. Each send writes a status record keyed by send id, transitioning through queued, submitted, delivered/failed, with the terminal state often arriving later via the provider’s receipt webhook (SMS/email) or known immediately (push, mostly). Because volume is enormous and the access pattern is “write once, read by recent time window / by recipient”, store status in a wide-column or time-series store with TTL, not a single hot relational table. This is what lets on-call answer “did the password reset actually arrive?” — and what powers product analytics (open rates, channel effectiveness) and the per-user rate limiter (it reads recent send history).

Bottlenecks & tradeoffs

The binding constraint is the providers, not your compute — they are rate-limited, flaky, and bill per message (SMS especially), so the whole design exists to insulate against them. Key tradeoffs:

• At-least-once + dedup vs exactly-once. Exactly-once across an external provider is unattainable (it may deliver then fail to ack). You choose at-least-once delivery and pay for an idempotency store; the cost is a Redis lookup per send, the payoff is no duplicate storms.
• Per-channel queues vs one queue. Separate queues isolate a slow provider to its own channel (push outage does not block email) and let each channel tune rate/retry independently — at the cost of more moving parts. Worth it: failure isolation is the point.
• Push-on-write vs topic broadcast. Per-recipient sends give per-user preference/personalization but cost N writes; provider topics give cheap broadcast but no per-user logic. The hybrid (topics for broadcast, per-recipient for personalized) is the standard answer.
• Status retention vs cost. Full delivery tracking at this volume is terabytes; you cap retention (for example 30 days hot, archive or drop after) and store in a write-optimized, TTL’d store rather than a relational table that would buckle.

The deepest tradeoff is priority: a transactional password reset and a bulk marketing blast share infrastructure but must not share fate. Separate priority lanes (queues) so the urgent, low-volume, must-arrive traffic is never stuck behind the bulk, best-effort flood.