Design nearby friends
Design real-time nearby-friends: ingest a flood of location pings (write-heavy), route updates through region-keyed pub/sub to subscribers over WebSockets, fan out only to people who care, and make privacy and TTL first-class so stale or unwanted locations expire.
The previous lesson’s coffee shops never move. Your friends do — constantly, all at once, all over the map. Flip the read-heavy proximity problem on its head and it becomes a write firehose: every phone pings its location every few seconds, and for each ping you must figure out, in real time, which of that user’s friends are close enough to care and push the update to their screens. The naive design — write each ping to a database, then have every client poll “where is everyone?” every second — collapses twice over: the writes saturate the database, and the polls multiply into a thundering herd. The interesting engineering is in not delivering most updates: a location ping should reach only the handful of people for whom it actually changed something.
By the time you finish this lesson you’ll understand why polling collapses under 2M writes per second, how a region cell becomes a routing key, and what makes privacy structural rather than a cleanup job.
Requirements
- Functional: a user shares their location; friends within some radius (say 5 km) see that user appear and watch them move in near-real-time. When a friend leaves the radius, they disappear. Sharing is opt-in and revocable.
- Non-functional: write-heavy — a ping every few seconds per active user. Low end-to-end latency (a couple of seconds, not minutes). Updates need not be durable; a dropped ping is replaced by the next one in seconds. Privacy is a hard requirement, not a feature: locations must expire and must never leak to non-friends.
The defining inversion from the proximity lesson: there, data was static and reads dominated. Here, data is in constant motion and writes dominate, while the “query” (who is near whom) must be answered continuously rather than on demand.
Estimation
Say 100M users, 10M active and sharing at any moment, each pinging every 5 seconds. That is 10^7 / 5 = 2×10^6 location writes per second — a firehose no single relational primary survives. Note what we do not need: durability or history. A location is interesting for seconds, then worthless. That single fact — ephemeral, high-volume, low-value-per-item — points straight at an in-memory, expiring store rather than a disk-backed database. The read side is bounded differently: each ping must fan out only to that user’s online friends within range, which for a typical social graph is single digits, not millions.
High-level design
Phones do not poll; they hold a persistent connection (a WebSocket) to a gateway, so the server can push updates the instant they happen. An incoming ping is written to a fast ephemeral store keyed by region cell, then published to a per-region pub/sub channel; subscribers interested in that region receive it and the gateway pushes it down the relevant sockets.
The two deep mechanisms are the persistent-connection layer that makes push possible, and the region-keyed pub/sub that keeps the fan-out from exploding.
Deep dive
WebSockets and the connection layer
Real-time push needs the server to send without the client asking, which rules out request/response polling. A WebSocket upgrades a normal HTTP connection into a full-duplex channel that stays open, so the gateway can stream updates down it with no per-message handshake. The architectural consequence is that the system is now stateful at the edge: 10M sharing users means 10M open connections, each pinned to a specific gateway process that knows which user and which sockets it holds. You scale this horizontally — many gateway nodes, a connection-aware load balancer, and a registry mapping user → gateway so a message for a user can be routed to the node holding their socket.
▸Why this works
Why not just poll every second? Two reasons, both about the asymmetry of cost. First, polling pays a full HTTP request/response (and often a TLS resumption) for every check, even when nothing changed — and the overwhelming majority of polls return “no change”, so you burn CPU and bandwidth manufacturing non-answers. Second, polling fixes your latency to the poll interval: a 1-second poll means up to 1 second of staleness even when an update was ready instantly, and shortening the interval to cut staleness multiplies the load. A persistent connection inverts both: the server sends exactly when there is news, the client pays connection cost once, and latency drops to network transit. The price is server-side state — millions of open sockets to manage — which is the trade you accept for real-time push.
Region-keyed pub/sub and the fan-out problem
The expensive mistake is broadcasting. If every ping went to every connection, 2M writes/sec times millions of connections is a quadratic blow-up that no cluster survives. The fix is to make location the routing key. Divide the world into region cells (the geohash/H3 idea from lesson 01, reused here for routing rather than search). Each ping is published to exactly one channel — its current region cell. A gateway subscribes only to the cells where it holds users who are sharing or watching. So a ping in downtown Berlin is delivered only to subscribers of Berlin-center cells, never to anyone in Tokyo.
This converts an all-to-all broadcast into a localized fan-out: the number of recipients for a ping is bounded by “online friends currently near that location”, which is small. As a user moves across a cell boundary, their client (or the gateway) re-subscribes to the new cell and unsubscribes from the old one — the same boundary awareness as lesson 01, now applied to a moving subscription rather than a static query. Redis pub/sub, a Kafka topic-per-region, or a dedicated message fabric all implement the same shape.
▸Common mistake
A tempting but wrong optimization is to fan out on write: when a ping arrives, look up the user’s friends, check each one’s last-known location, and push directly to the nearby ones. It seems to skip the pub/sub layer. It fails at scale because it does the expensive graph + geo join on the hottest path (2M times/sec), couples the ingest service to the social graph and presence, and re-does that work for every single ping even when nobody moved meaningfully. Region-keyed pub/sub decouples the firehose from the friend-matching: ingest just publishes to a cell, and the relatively cheap “is this publisher my friend, and still in range” filter runs at the subscriber edge, on a stream that has already been narrowed to one region. Push the filtering to where the data is already small.
Privacy and TTL
Privacy cannot be a layer you add later; it shapes the data model. Every stored location carries a TTL (time-to-live) of a few seconds to a minute, so a phone that goes dark — app closed, battery dead, sharing revoked — simply ages out of the store with no explicit delete. This gives “last seen a moment ago” semantics for free and guarantees stale locations do not linger as a privacy hazard. On top of TTL, three rules: sharing is opt-in and per-relationship (I may share with Alice but not Bob); the fan-out filter enforces friendship so a region subscriber never receives a non-friend’s coordinates (region channels carry pings, but the edge drops any whose publisher is not an authorized friend of the recipient); and the server stores coarse precision where the product allows, since “within this neighborhood” leaks far less than exact GPS.
Bottlenecks & tradeoffs
The headline bottleneck is connection state, not throughput of any single message. Millions of long-lived sockets are memory and file-descriptor heavy, survive deploys awkwardly (a rolling restart drops every connection unless you drain gracefully), and demand a routing registry that itself must scale. The headline tradeoff is freshness vs cost: pinging every second feels live but quadruples write load over every-five-seconds for marginal benefit, so adaptive ping rates (slower when stationary, faster when moving) buy most of the realtime feel for a fraction of the writes. And the deepest tradeoff is the one the whole design rests on — we deliberately drop durability and history to make the volume survivable; if the product later wants “where was my friend an hour ago”, that is a separate, durable, batch-written store, never the live path.
A nearby-friends design ingests 2M location pings/second. The team writes each ping to the primary SQL database and has clients poll 'where are my friends?' once per second. It falls over immediately. What are the two structural fixes?
To deliver a location ping to the right people, an engineer proposes: on each ping, load the publisher's friend list, fetch each friend's location, and push directly to the nearby ones. Why is region-keyed pub/sub preferred at 2M pings/sec?
Stored locations carry a short _______ so that a phone which goes dark — app closed, sharing revoked — simply ages out of the store with no explicit delete, giving 'last seen a moment ago' semantics and ensuring stale positions never linger as a privacy hazard.
- 01Why is nearby-friends write-heavy, and what storage and delivery model does that force?
- 02How does region-keyed pub/sub solve the fan-out problem?
- 03How is privacy made structural, and what are the main bottlenecks and tradeoffs?
Nearby-friends is the inverse of proximity search: locations move constantly and writes dominate (about 2M pings/sec for 10M sharers), but each location is ephemeral — no durability or history needed. That forces an in-memory, TTL-expiring store instead of a database, and WebSocket push instead of polling, so the server sends the instant there is news at the cost of millions of stateful edge connections. The fan-out is tamed by making location the routing key: each ping is published to exactly one region-cell channel, gateways subscribe only to cells where they hold users, and the cheap “friend and in-range” filter runs at the subscriber edge — never as a graph+geo join on the ingest firehose (the fan-out-on-write trap). Privacy is structural: TTL auto-expiry, opt-in per relationship, a friendship filter at the edge, and coarse precision where allowed. The bottleneck is connection state, not message throughput; the key tradeoffs are freshness vs cost (adaptive ping rates) and the deliberate sacrifice of durability and history to make the volume survivable. Now when you see a “live location” feature in a design review, you’ll know to ask: is the storage ephemeral, are connections pushed not polled, and does the fan-out filter run at the subscriber edge?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.