Publish/subscribe

From one-to-one to one-to-many

A point-to-point queue delivers each message to one consumer. Publish/subscribe delivers each message to every interested party. The publisher emits a message to a named topic; the broker copies it to each subscription; each subscriber consumes its own copy independently. The publisher names a topic, not a recipient — it has no idea who is listening, and consumers come and go without the publisher knowing or caring. That indirection is the whole point: it adds a level of decoupling beyond the temporal decoupling of a plain queue.

The result is the inversion in the hook. Instead of the producer holding an N-long list of downstreams to call (and a deploy whenever that list changes), the producer publishes once and the consumers decide what they care about. This is the difference between commands (“send this email” — addressed to one handler, a job → use a queue) and events (“an order was placed” — a fact, broadcast to whoever cares → use pub/sub). Getting that distinction right is most of the design: a command has exactly one correct handler; an event has zero or many, and you must not encode the consumers into the producer.

Two architectures wearing the same name: broker vs log

“Pub/sub” describes a behaviour (fan-out), and two very different mechanisms implement it. The split is the most important thing to understand in this lesson.

Broker-style (RabbitMQ, ActiveMQ, AWS SNS→SQS). The broker routes a copy of each message into a per-subscriber queue, and once a subscriber acknowledges, that copy is deleted. Message storage is transient — the broker’s job is delivery, not retention. A subscriber that wasn’t connected when a message was published may simply never see it (unless you configured durable subscriptions). The broker tracks delivery state on behalf of consumers.

Log-based (Apache Kafka, AWS Kinesis, Apache Pulsar). The topic is an append-only log: messages are written in order and retained for a configured window (hours, days, or forever), independent of whether anyone has read them. Consumers don’t get messages pushed and deleted; they pull and track their own position — an offset — into the log. Multiple independent consumers read the same log at their own offsets without interfering. Reading does not consume; the log just remembers where each consumer is.

BROKER (RabbitMQ/SNS)            LOG (Kafka/Kinesis)
publish → fan copies → queues    append → immutable log [m0 m1 m2 m3 m4 …]
consumer acks → copy DELETED      consumer A reads, advances its OFFSET → 3
state lives in the BROKER         consumer B reads independently → offset 1
"a delivery pipe"                 state (offset) lives in the CONSUMER
                                  "a durable, replayable record"

This single architectural choice cascades into everything else: replay, ordering, throughput, and operational model all follow from “delete on consume” versus “append-only log with offsets.”

Consumer groups and partition ordering

A log topic is split into partitions for parallelism — the unit of both scale and ordering. A consumer group is a set of cooperating consumers that share the work of a topic: each partition is assigned to exactly one consumer in that group, so adding consumers up to the partition count scales throughput linearly. Crucially, different groups each get the full stream — group “analytics” and group “fraud” both read every message at their own offsets. That is how one topic feeds many independent subscribers (fan-out across groups) while each subscriber load-balances internally (sharing across the group).

Ordering follows from partitions. The log guarantees order within a partition, not across the topic. Messages with the same key (e.g. user_id) hash to the same partition, so all of one user’s events stay ordered relative to each other; events of different users may interleave. This is the same per-key ordering tradeoff from the queues lesson, now made structural: you get parallelism across partitions and ordering within one, and you choose your partition key to put “things that must stay ordered” together. Pick a key with too few distinct values and you create a hot partition that can’t be parallelized.

So is a log just a fancy queue? No.

The deepest takeaway: a queue and a log are not the same tool with different branding. A queue is a delivery pipe — messages flow through and are gone once handled; its state is “what’s left to do.” A log is a durable record of facts — messages stay, and consumers move over them; its state is “where is each reader.” A queue answers “what work remains?”; a log answers “what happened, and can I read it again?” You reach for a broker/queue when work should be done once and forgotten, and for a log when events are facts you may want to fan out, reorder consumers over, or replay. Confusing them — treating Kafka as a task queue you delete from, or a queue as an event store you replay — is a common and expensive mismatch.

This is composition altitude: choosing fan-out and choosing log-vs-broker for a design. The dedicated queues track drills the internals — Kafka partition leadership and the ISR, RabbitMQ exchange types and bindings, exact offset-commit protocols. Use it when you’re operating a specific broker; stay here when you’re deciding whether the design wants fan-out and whether it wants replay.