Queues, Streams, Eventing
Message ordering: build an order-surviving consumer
Reading about ordering hazards is not the same as watching a debit land before its credit and proving your design absorbs it. Build a small event pipeline, inject the disorder production will throw at it — concurrency, producer retries, redelivery, DLQ replay — and show the consumer stays correct, with evidence at every step.
Turn the unit’s mental model into a reproducible engineering loop: choose the right partition key, harden the producer, make the consumer idempotent and version-guarded, then deliberately break ordering and prove the final state is still correct.
Build an account-balance (or profile-update) consumer over Kafka or SQS FIFO that processes per-entity events concurrently and provably converges to the correct final state even under reordering, duplicate delivery, and DLQ replay — with a test harness that demonstrates it.
- A run log showing concurrent consumers processing multiple entities in parallel, with one entity's events kept in version order despite out-of-order arrival.
- Evidence the version guard fired: counts of events applied vs skipped (stale/duplicate), and the final per-entity state matching the in-order expected state.
- A forced-retry or duplicate-delivery test that would reorder or double-apply on a naive consumer but leaves your final state correct.
- A short write-up naming the partition key you chose and why, plus which mechanism (partitioning, idempotence, version guard) caught each class of disorder.
- Trigger a repartition (grow Kafka partitions, or change the keying) and show the transient ordering corruption, then show your version-guarded consumer still converges to the correct state.
- Add an on-call runbook: how to spot per-key reordering in metrics, the producer/consumer config checklist, and the safe DLQ-replay procedure (rate-limited, version-guarded).
- Replace the version guard with commutative / last-write-wins effects (set balance = X instead of add X) and compare which class of operations each design tolerates.
- Run the same workload on the other broker (Kafka if you started on SQS FIFO, or vice versa) and compare how each expresses per-key order and what throughput ceiling a single key hits.
This is the loop you will run on every real ordering design: pick the partition key as the consistency boundary, harden the producer so it cannot reorder at the source, make the consumer idempotent and version-guarded so redelivery and DLQ replay are no-ops, then deliberately inject disorder and prove the final state still converges. Doing it once on a toy pipeline — and watching the version guard reject a stale debit — makes the production version muscle memory.