Design a news feed

Requirements

When you write down requirements here, you are really answering one question: where does the expensive work live — on the write path or the read path? Everything else follows from that answer.

Functional: a user posts content; their followers see it in a feed. A user opens their feed and gets a ranked, paginated list of recent posts from accounts they follow. Support infinite scroll. Posts can be liked/commented (which feeds back into ranking). The follow graph is asymmetric (you follow accounts that may not follow you).

Non-functional: the system is massively read-heavy — feed reads dwarf posts by orders of magnitude, so the design optimizes the read path above all. Feed load must feel instant (tens of milliseconds for the first page). New posts should appear “soon” (seconds to a minute is fine — eventual consistency is acceptable here; nobody’s bank balance is wrong if a post shows up 20 seconds late). It must handle the power-law follower distribution: most accounts have few followers, a tiny number have tens of millions, and that long tail is the entire design problem.

The defining tension: do you do the expensive work when a post is written (and make reads cheap) or when a feed is read (and make writes cheap)? That single choice is the heart of feed design.

Estimation

300M daily active users, each opening their feed ~10 times/day, gives 3×10^9 feed reads/day, about 35,000 feed reads/second average, peaking several times higher. Posts are far rarer — say each user posts ~twice/day, so ~7,000 posts/second. The read:write ratio is roughly 100:1, which is the single most important number: it justifies precomputing feeds (do work once on write, serve it cheaply on the 100 reads).

Now the fan-out cost. Average followers might be a few hundred, so an average post fans out to a few hundred feed writes — fine. But a celebrity post with 30M followers, under pure fan-out-on-write, is 30M writes for one post. If even a handful of celebrities post in the same minute, that is hundreds of millions of writes hammering the feed store — the hook’s outage. The napkin says: fan-out-on-write is cheap on average and catastrophic at the tail, so the tail needs different handling.

High-level design

The two strategies are mirror images; the production system runs both.

• Fan-out-on-write (push): when a post is created, immediately insert its id into the precomputed feed of every follower. Reads are then a trivial cache lookup. Cost: write amplification proportional to follower count.
• Fan-out-on-read (pull): store posts only once (by author); when a user reads their feed, query the recent posts of everyone they follow and merge. Writes are trivial. Cost: every read does a scatter-gather across many authors — expensive, and repeated on every refresh.
• Hybrid: push for normal authors (cheap, common), pull for celebrities (avoids the write storm), merge at read time.

Deep dive

Fan-out-on-write vs fan-out-on-read

This is the core tradeoff, and it is the read:write ratio made concrete. With a 100:1 read:write ratio, doing work on write and amortizing it over 100 reads is usually the win — which is why push is the default. The precomputed feed is just a per-user list of post ids (a Redis sorted set, scored by time or rank); a read is “fetch the top N ids, hydrate the posts” — milliseconds.

Pull is the opposite bet. It is write-cheap (one copy of each post) but read-expensive: assembling a feed means querying the recent timeline of every followed account and merging them, on every single refresh, with no amortization. For a user following 1,000 accounts, that is a 1,000-way scatter-gather per feed load. Pull wins only when writes vastly outnumber reads or when followers rarely read — the opposite of a feed.

So why not push always? Because of the tail.

The celebrity problem and the hybrid

Push breaks for high-follower accounts. A 30M-follower post is 30M writes; the “hot key” of celebrities means a few accounts generate a disproportionate share of all fan-out work, and a burst of celebrity posts is the write storm from the hook. The elegant fix is to invert the strategy for the few accounts that break the common one: do not fan out celebrity posts on write at all. Instead, store them once, and at read time, for each celebrity a user follows, pull that celebrity’s recent posts and merge them into the (mostly precomputed) feed.

This works because the asymmetry runs both ways: celebrities are few (so the per-read pull is a handful of extra lookups, not thousands), and they are followed by many (so push would have been catastrophic). A threshold (say, accounts over ~100k–1M followers) classifies an author as push or pull. The read path becomes: read the precomputed feed, pull recent posts from followed celebrities, merge, rank, slice. The merge is cheap because celebrities-followed-per-user is small.

Ranking, feed cache, and pagination

Ranking. A modern feed is not strictly reverse-chronological; it is ranked by predicted engagement (recency, affinity to the author, content type, predicted like/comment probability). The pragmatic design keeps a candidate-generation then ranking split: fan-out/pull produces a candidate set of recent post ids (cheap), and a ranking stage scores them (more expensive, applied to the small candidate set, not the whole graph). Ranking can run at read time over the candidates or be partially precomputed; either way it operates on a bounded candidate list, never the whole timeline.

Feed cache. The precomputed feed lives in an in-memory store (Redis sorted set per user) holding the top few hundred post ids — not full posts. You cap the list (you do not store a user’s entire history in cache; old entries fall off), and on a cache miss or for inactive users you can lazily rebuild from the authors’ timelines. Storing ids, not bodies, keeps the cache small and lets the same post object be hydrated once and shared. The hot-key risk (a viral post) is handled by caching the post body itself behind a CDN/edge cache so the hydration of a popular post does not stampede the post store.

Pagination. Never offset-paginate a feed. Offset pagination (LIMIT 20 OFFSET 10000) gets slower as you scroll and skips or duplicates items when new posts arrive between requests. Use cursor (keyset) pagination: the client sends the score/id of the last item it saw, and the server returns items after that cursor. It is O(1) regardless of depth and stable under inserts — the right primitive for infinite scroll over a constantly-growing list.

Timeline storage

Two things are stored separately. Posts (the source of truth) live in a durable store sharded by author id — author timelines are written once and read by the pull path and by hydration. Feeds (the derived, per-consumer view) live in the feed cache, sharded by consumer/user id, holding ranked post ids with a cap and TTL. Keeping these distinct is what lets the same post be referenced by millions of feeds without copying its body, lets the feed be rebuilt from posts on demand, and lets you shard the write-heavy author timelines and the read-heavy consumer feeds on different keys for their different access patterns.

Bottlenecks & tradeoffs

The dominant force is the power-law follower distribution: it is why no single strategy works and why the hybrid exists. The key tradeoffs:

• Write cost vs read cost. Push pays on write to make reads cheap (right for read-heavy); pull pays on read to make writes cheap (right only when reads are rare). The read:write ratio decides the default, and the follower distribution decides the exceptions.
• Freshness vs cost. The feed is eventually consistent — a post appearing seconds late is fine, and accepting that staleness is what lets you precompute, cache, and avoid re-fanning-out on every mutation. Demanding strict freshness would force pull (expensive reads) or constant re-fan-out (expensive writes).
• Storage duplication vs assembly cost. Push duplicates a post id into many feeds (storage + write amplification) to make reads O(1); pull stores once but pays assembly per read. Storing ids (not bodies) bounds the duplication cost.
• Ranking quality vs read latency. Heavier ranking improves engagement but costs read-time compute; the candidate-generation/ranking split bounds that cost by ranking only a small candidate set, not the whole graph.

The deepest insight: the feed cache is a derived, approximate, eventually-consistent index, not the source of truth. Treat it as authoritative and every edit becomes a fan-out storm; treat it as a fast candidate list resolved against the real posts at hydration, and the system stays cheap and correct.