Design a distributed message queue

Requirements

The hook’s fraud team didn’t need a faster queue — they needed one with a different contract. When you read the functional requirements below, notice which one the classic broker violated and which requirement makes replay and multiple consumers structurally free rather than bolted on.

Functional

• Produce: append a message to a named topic.
• Consume: read messages from a topic, in order, starting from a chosen position.
• Multiple independent consumers: several different systems read the same topic without interfering (the fraud scorer, the ledger, the analytics pipeline all read the payment stream).
• Replay: a consumer can rewind and re-read past messages (the hook’s missing feature).
• Retention: messages are kept for a configured window (time or size), not deleted on delivery.

Non-functional

• High throughput. Millions of messages/sec — this is a firehose, not a mailbox.
• Durability. An acknowledged message must survive broker crashes; losing a payment event is unacceptable.
• Horizontal scale. Add brokers to add capacity; no single broker is a ceiling.
• Ordering guarantee — at least within a partition.
• Tunable delivery semantics. Different consumers tolerate different failure modes (a metrics pipeline can drop a message; a ledger cannot).

The animating idea is that the queue is a log, not a mailbox: append-only, retained, and read by position. Everything else is how to make that log fast, durable, and parallel.

Estimation

• Target 1 million messages/sec at peak, average message 1 KB → ~1 GB/sec of write throughput. This is the headline number and it rules out any single-broker design.
• Retention of 7 days: 1 GB/s × 86,400 s × 7 ≈ ~600 TB of log on disk (before replication). With replication factor 3, ~1.8 PB. The log lives on disk, not RAM — so sequential disk I/O performance is the whole ballgame.
• Per-partition throughput caps at maybe ~10 MB/sec for ordered, replicated writes, so 1 GB/sec needs on the order of ~100+ partitions spread across brokers — partition count is how you buy throughput.
• Consumers: if reads are mostly sequential from the tail of the log, the OS page cache serves recent messages from RAM, so a healthy cluster does almost no random disk reads — the access pattern is what makes the numbers work.

The key insight from the numbers: sequential append to disk plus partitioning is what turns a “slow” disk into a 1 GB/sec firehose, because sequential disk throughput rivals memory bandwidth while random I/O would be ~1000× slower.

High-level design

A topic is split into partitions; each partition is an ordered, append-only log living on a leader broker and replicated to follower brokers. Producers append to partitions (keyed or round-robin). Consumers in a consumer group divide the partitions among themselves and track an offset per partition.

Deep dive

Partitions: the unit of parallelism and ordering

A topic is divided into partitions, and the partition is the atom of the whole system. Three things follow from it:

• Parallelism. Different partitions live on different brokers and are written/read independently, so throughput scales with partition count — this is how you get from one broker’s limit to 1 GB/sec.
• Ordering is per-partition, not per-topic. Messages within a partition are strictly ordered by offset; across partitions there is no global order. So if you need ordered processing of a user’s events, you must route all of that user’s messages to the same partition — done by hashing a partition key (e.g. user_id). No key → round-robin → maximum spread but no ordering guarantee.
• Consumer parallelism is capped by partitions. Within a consumer group, each partition is consumed by exactly one consumer, so you can have at most as many active consumers as partitions. Partition count is therefore both your throughput knob and your max consumer parallelism — chosen up front and awkward to change later.

Taken together, the partition is a single knob that buys you throughput, ordering, and consumer parallelism simultaneously — but only if you choose the partition key right. A wrong key (or none) and you lose ordering where you need it and create a hot partition that the rest of the cluster cannot help with.

Replication and the ISR

Each partition has one leader (takes all reads and writes) and several followers that replicate the leader’s log. Durability hinges on the in-sync replica (ISR) set: the followers that are currently caught up with the leader. A write is considered committed only once all ISR members have it, and acks controls how strict the producer is:

• acks=0 — fire-and-forget; fastest, can lose data.
• acks=1 — leader-only ack; loses data if the leader dies before followers replicate.
• acks=all — every ISR member must have the message before it’s acknowledged; survives leader failure with no loss, as long as the ISR isn’t empty.

When the leader dies, a new leader is elected from the ISR, so it already has every committed message — no committed write is lost. The tunable min.insync.replicas sets how many replicas must be in-sync for an acks=all write to succeed; set it to 2 (with replication factor 3) and you survive one broker failure while still rejecting writes if durability would be compromised.

Offsets, consumer groups, and delivery semantics

Because the broker retains the log and never deletes on delivery, each consumer simply tracks an offset — the position of the next message to read — and commits it periodically. This is the engine behind three features:

• Replay: reset the offset backwards and re-read (the hook’s fix — replay the fixed code over old messages).
• Multiple consumer groups: each group has its own offsets, so the fraud scorer, ledger, and analytics each read the full stream independently at their own pace.
• Failure recovery: a crashed consumer resumes from its last committed offset.

But when you commit the offset relative to when you process the message defines the delivery semantics — the single most important correctness decision in any async system:

• At-most-once: commit the offset before processing. If the consumer crashes mid-process, the message is skipped on restart — possible loss, no duplicates. (Acceptable for metrics.)
• At-least-once: commit the offset after processing. If the consumer crashes after processing but before committing, the message is re-read and reprocessed — possible duplicates, no loss. (The common default.)
• Exactly-once: no duplicates and no loss. Genuinely hard — it requires either idempotent processing (the consumer dedupes by message ID so reprocessing is harmless) or a transaction that atomically commits the offset and the side effect together. Kafka offers transactional producers/consumers for this, but most systems achieve “effectively once” with at-least-once delivery plus an idempotent consumer.

Bottlenecks & tradeoffs

• Partition count is a one-way-ish decision. Too few and you cap throughput and consumer parallelism; too many and you pay coordination, file-handle, and rebalance overhead. Increasing partitions later breaks key-based ordering (a key may hash to a new partition), so you size generously up front.
• Hot partition / skewed key. A bad partition key (e.g. keying by a country where one country dominates) sends most traffic to one partition and one consumer — the partitioned log’s version of a hot shard, and the whole cluster can’t help.
• At-least-once means consumers must be idempotent. The common default delivers duplicates on retry, so every consumer that has side effects (charge a card, send an email) must dedupe by message ID — a requirement teams forget until they double-charge someone.
• Rebalance storms. When a consumer joins or leaves a group, partitions are reassigned and consumption pauses; frequent membership churn (flapping consumers, long GC pauses) causes repeated rebalances that stall the group — so consumer liveness tuning is operationally important.
• Retention vs cost. Long retention enables replay but costs petabytes; the window is a direct money-vs-recoverability trade, and “infinite retention for replay” is a real bill.