Async & messaging: an event-driven order pipeline
Hands-on project: build a small event-driven order pipeline with an idempotent at-least-once consumer, a transactional outbox, and a bounded queue, then break it on purpose — kill the relay, redeliver, and overload it — to prove your guarantees hold.
Reading that the outbox defeats the dual-write problem is not the same as watching your pipeline survive a relay crash, a redelivery storm, and an overload — and stay correct. Build a small event-driven order pipeline, then deliberately break it: kill the process between commit and publish, redeliver every message twice, and bury the consumer in load. The point isn’t a polished service; it’s seeing your guarantees hold (or fail) under the exact crashes this unit warned you about.
This project makes the whole unit operational: you’ll implement the patterns — at-least-once with idempotency, the transactional outbox, a bounded queue with backpressure — and then run the failure injections that separate code that looks correct from code that is correct under crashes.
Build a minimal event-driven order pipeline — an order service that writes an order and emits an OrderPlaced event via a transactional outbox, a queue/log carrying the event, and one or more idempotent consumers (e.g. inventory, email) — then inject the failures from this unit (relay crash, redelivery, overload) and demonstrate that no event is lost, no effect is duplicated, and the system degrades gracefully under load.
- A working pipeline where placing an order reliably results in exactly-effectively-once downstream effects, demonstrated end to end.
- Failure injection A: a recorded run (logs/screens) where the process is killed in the commit→publish gap and, after restart, the relay publishes the pending outbox row — proving 'never lost' and that the outbox, not luck, provides it.
- Failure injection B: a recorded run where every message is delivered twice and the consumer's dedup store shows the second delivery skipped — proving the effect happened once despite at-least-once delivery.
- Failure injection C: a recorded run where sustained overload triggers backpressure (rejections/shedding or bounded blocking) instead of an ever-growing backlog, with a queue-depth/lag plot and the chosen bound stated.
- A short write-up: the delivery semantic you started from, exactly where the outbox restores atomicity, why the consumer is idempotent, and what your bound + DLQ + lag alarm protect against — plus one paragraph on the eventual-consistency window a user would experience and how you'd surface it in a UI.
- Add a second consumer group (analytics) on the same stream and demonstrate fan-out — both groups see every event independently — and, on a log, replay the analytics consumer from offset 0 to backfill.
- Add a retry storm: enable aggressive client retries during overload and show the queueing collapse accelerate, then tame it with exponential backoff + jitter and a circuit breaker, comparing the two backlog curves.
- Turn the flow into a small orchestration (saga): coordinate order → charge → reserve with a coordinator that runs a compensating action (refund/release) when a later step fails, and show the process state is queryable ('where is order #X?').
- Add multi-tenant fairness: shuffle-shard tenants across a small pool of queues and show that one tenant's spike no longer starves the others.
- 01What does each of the three failure injections prove, and why inject them?
- 02Where exactly does the outbox restore atomicity, and what must the consumer still do?
This project turns the async unit into a repeatable workflow: build a minimal event-driven order pipeline — an order service that writes the order and an outbox row in one local transaction, a relay that publishes OrderPlaced from the outbox, a broker/log, and an idempotent at-least-once consumer that dedups on order_id — then make every guarantee earn its keep by breaking the system on purpose. Injection A kills the process in the commit→publish gap and shows the relay still publishes after restart (the outbox = never lost). Injection B delivers every message twice and shows the dedup store skip the second (idempotency = once-effect). Injection C overloads the consumer and shows a bounded queue apply backpressure instead of an unbounded backlog, with a DLQ for poison messages and a lag alarm as early warning. The write-up ties it together: the delivery semantic you started from, where atomicity is restored, why the consumer is idempotent, and how you’d surface the eventual-consistency window in a UI. An engineer who has built and broken this once designs async systems for the crash, not just the happy path — which is the entire point of this unit.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.