Queues, Streams, Eventing
Message ordering: total order is a throughput tax, partial order is the deal
A wallet service emits balance.credit then balance.debit for the same account. In staging, with one consumer, it works for months. In prod you scale to eight consumers for throughput and the debit lands first: a customer briefly goes negative, a fraud rule trips, the account is frozen. Nothing in the code changed. You just removed the accidental single-threaded ordering that staging gave you for free, and discovered the queue never promised order across consumers in the first place.
Total order has exactly one shape, and it is slow
“Process everything in the exact order it was produced” sounds like a config flag. It is not. The only way to get a single global sequence is a single serialization point: one partition, written by one producer ordering, drained by one consumer at a time. The moment two consumers can run in parallel, there is no longer a single timeline — message 5 and message 6 race, and whichever handler finishes first wins. Total order and parallelism are physically opposed: the ordering you want is a queue of one.
That is the throughput tax. SQS FIFO with a single message group is capped at 300 messages/second (3,000 with batching of 10), because everything in one group is delivered one-at-a-time — SQS will not hand you the next message in a group until the previous one is deleted. Kafka with one partition is limited to that single partition’s throughput and a single consumer in the group; adding consumers does not help because a partition is assigned to exactly one consumer. You bought total order with your scalability.
Partial order is what you actually want
The senior insight is that you almost never need total order across everything — you need order within a thing. The wallet does not care whether account A’s credit is ordered relative to account B’s debit; those are independent. It cares intensely that A’s own events stay in sequence. That is partial order: messages sharing a key are FIFO with respect to each other, while different keys parallelize freely.
This is the model both major systems give you:
- Kafka orders within a partition. Choose a partition key (the account id) and Kafka hashes it to a partition; all events for that key land in the same partition, in order, at full per-partition throughput. Different keys spread across partitions and consumers.
- SQS FIFO orders within a
MessageGroupId— ordering is strictly per group, never across groups. Same group = strict FIFO and one-at-a-time delivery; different groups process concurrently. Aggregate throughput therefore scales with the number of message groups: high-throughput mode reaches tens of thousands of messages/second only by spreading load across many groups and batching, not by speeding up any single group.
Pick the key as the entity that must stay consistent — per-account, per-aggregate, per-user — and you get order exactly where it matters and parallelism everywhere else.
| Model | Serialization point | Throughput | Use when |
|---|---|---|---|
| Total order | One partition / one consumer | 1 group: ~300 msg/s (SQS), one partition (Kafka) | Rare — a single global ledger that truly must serialize |
| Partial order (per-key FIFO) | One per key/partition/group | Scales with key count; full per-partition throughput | Almost always — order within an entity, parallel across entities |
| No order | None | Maximum — every consumer fully independent | Commutative/idempotent work where order is irrelevant |
How ordering silently breaks in production
Partial order is not free — it holds only while three things stay true, and prod loves to break each one.
1. You add partitions. Kafka maps a key to a partition by hashing modulo the partition count. Increase partitions from 6 to 12 and the same key now hashes to a different partition. In-flight events for that key are still being drained from the old partition while new ones pile into the new one — for a window, two partitions hold the same key’s events and two consumers process them concurrently. Repartitioning is the classic ordering-corruption event; it is why teams over-provision partitions up front rather than grow them.
2. Retries and the DLQ reinsert messages out of band. A handler fails on message 5, the message goes to a dead-letter queue, and later someone replays it — but messages 6, 7, 8 already committed. Now 5 lands after its successors. Even within a single partition, a retry that re-enqueues to the tail reorders against everything produced in the meantime. DLQ replay at full speed also reorders a whole batch against live traffic, which is why you replay rate-limited, not all at once.
3. The producer reorders under retry. This one bites at the source. With Kafka, if enable.idempotence is false and max.in.flight.requests.per.connection is greater than 1, a batch that fails and retries can land after a later batch that succeeded — silently swapping two messages on the same partition. KIP-185 made the idempotent producer the default precisely to close this: the broker tracks a per-producer, per-partition sequence number, drops duplicates, and refuses out-of-sequence batches, preserving order even with up to 5 in-flight requests.
Why this works
“At-least-once delivery” and “ordered” are independent properties, and most queues give you at-least-once. That means a message can be delivered twice and a redelivery can arrive after newer messages. So even a correctly partitioned per-key FIFO stream can show you [5, 6, 5] after a redelivery of 5. Ordering guarantees describe the happy path; the duplicate-and-reorder path is the one your handler must survive.
The patterns that make order survivable
You cannot make a distributed system perfectly ordered cheaply, so seniors design so that occasional disorder is harmless. Four moves, roughly in order of leverage:
- Partition by the entity that needs order. Choose the key as the consistency boundary —
accountId, aggregate id,userId. This is the foundation; the other three handle the gaps it leaves. - Make handlers idempotent. Persist the message id (or an idempotency key) and skip anything seen before. Now at-least-once redelivery is a no-op instead of a double-charge, and DLQ replay is safe.
- Make effects commutative where you can. If the operation is
set balance = Xrather thanadd X, order stops mattering — the last write wins and re-applying is harmless. Commutative or last-write-wins effects turn an ordering problem into a non-problem. - Carry a sequence number / version and drop stale updates. Stamp each event with a monotonic version per entity; the consumer remembers the highest version applied and discards anything lower. An out-of-order or replayed update is detected and dropped. This is single-flight-per-key plus a version check — the strongest defense when effects are not commutative.
A profile service consumes user.updated events. Concurrent consumers + an at-least-once queue mean updates for one user can arrive out of order or twice. Pick the design.
Why does scaling a Kafka topic from one consumer to eight break the ordering you saw in staging?
With a Kafka producer, which combination can silently reorder two messages on the same partition?
Order the senior decisions for designing an ordered consumer, strongest leverage first:
- 1 Decide the consistency boundary: what entity must its events stay ordered within (account, aggregate, user)?
- 2 Partition / set MessageGroupId by that key so its events are per-key FIFO
- 3 Make the handler idempotent by message id so at-least-once redelivery and DLQ replay are safe
- 4 Prefer commutative / last-write-wins effects so residual disorder is harmless
- 5 Stamp a per-entity version and drop any update older than the last applied
- 01A teammate says 'just turn on FIFO and we get ordered processing.' Explain what FIFO actually buys and what it costs.
- 02Why does adding partitions to a Kafka topic risk corrupting per-key ordering, and how do teams avoid it?
Global total order has exactly one shape — a single serialization point, one partition drained by one consumer — and it caps throughput at roughly 300 messages/second on an SQS FIFO group or a single Kafka partition. You almost never need it. What you need is partial order: per-key FIFO, where messages sharing a key (account, aggregate, user) stay in sequence while different keys parallelize across partitions and consumers. That order is fragile in production: it breaks when you add partitions and a key rehashes, when retries or DLQ replays reinsert a message behind newer ones, and when a non-idempotent producer with multiple in-flight requests lets a retried batch overtake a later one — which is why the idempotent producer with per-partition sequence numbers became Kafka’s default. The durable design assumes occasional disorder and survives it: partition by the entity that needs order, make handlers idempotent so redelivery is a no-op, prefer commutative or last-write-wins effects, and stamp a per-entity version so out-of-order updates are detected and dropped. Order where it matters, parallelism everywhere else.