Event-driven architecture
Event-driven systems coordinate via facts, not calls — choreography vs orchestration. The load-bearing pattern is the outbox: it makes 'write the DB and publish the event' atomic, defeating the dual-write problem at the cost of eventual consistency you must show in the UX.
The order service did two things on checkout: it wrote the order to its database, then it published an “order placed” event so inventory and shipping could react. It worked in testing every time. In production, once every few thousand orders, the database commit succeeded and then the process died before the event was published. The order existed; the world never heard about it. Inventory wasn’t decremented, no shipping label was created, and a paying customer’s order silently vanished into a state no service knew to act on. The bug wasn’t in either step. It was the gap between them — two systems updated with no transaction spanning both. That gap is the dual-write problem, and the fix is the heart of this lesson.
Coordinating with facts instead of calls
In a request-driven system, service A makes service B do something by calling it: A holds the control flow, waits, and depends on B being up. In an event-driven architecture (EDA), A instead emits a fact — “an order was placed” — to a topic and moves on; whoever cares reacts on their own schedule. The control flow inverts: the producer no longer commands downstreams, it announces what happened, and the system’s behaviour emerges from independent reactions to those facts.
The payoff is the loose coupling from the pub/sub lesson, taken to the architecture level: services don’t know about each other, you add capabilities by adding subscribers, and one slow service doesn’t block the others. The cost is that you give up the simple, traceable control flow of “A called B called C.” Behaviour is now distributed across reactions, debugging spans services, and — the theme of this whole unit — you inherit asynchrony’s consequences: duplicates, ordering, and eventual consistency.
Choreography vs orchestration
A business process usually needs several steps (place order → charge → reserve stock → ship). There are two ways to coordinate them, and choosing wrong scales badly.
Choreography — no central brain. Each service listens for events and emits its own: the order service emits OrderPlaced; payments hears it, charges, emits PaymentTaken; inventory hears that, reserves stock, emits StockReserved; shipping reacts. The logic is the sum of the reactions, spread across services. It’s maximally decoupled and great for simple flows — but for a complex process the end-to-end logic exists nowhere: no single place says “this is what placing an order does,” which makes the flow hard to see, change, and debug.
Orchestration — a central coordinator (an “orchestrator” or saga) explicitly drives the steps: it tells payments to charge, waits for the result, tells inventory to reserve, and on a failure runs compensating actions (refund, release stock) to unwind. The flow lives in one readable place, which is far better for complex, multi-step, must-not-half-complete processes — at the cost of reintroducing a component that knows about everyone (though it commands via events/commands, not synchronous chains).
The rule of thumb: choreography for simple, few-step reactions where decoupling matters most; orchestration when the process is complex, has compensations, or someone needs to answer “where is order #8842 in the flow right now?”
▸Why this works
Why not just keep choreographing as the process grows? Because choreography distributes the logic but provides no place for the state of the whole process. With five services each reacting to the previous one’s event, no component knows whether order #8842 finished step 3 or stalled, and a failure mid-flow leaves orphaned partial state with no one responsible for cleanup. You end up reconstructing the flow from logs across five services to answer one question. An orchestrator centralizes that process state and the failure/compensation logic deliberately — you trade some coupling for the ability to see, resume, and unwind a long-running transaction. The mistake is treating “choreography = always more decoupled = always better”; past a few steps, the missing process state becomes the dominant cost.
Event sourcing and CQRS, briefly
When you’re designing an event-driven system, you’ll quickly run into two patterns that sound architectural but are often reached for prematurely — worth knowing so you can recognise when they’re justified and when they’re not. Two patterns ride along with EDA and are worth recognizing. Event sourcing: instead of storing current state and overwriting it, you store the sequence of events as the source of truth (AccountOpened, Deposited 100, Withdrew 30) and derive current state by replaying them. You gain a perfect audit log and the ability to rebuild any view — and this is exactly why a retained, replayable log (from the pub/sub lesson) is the natural substrate. CQRS (Command Query Responsibility Segregation): split the write model (commands, normalized for correctness) from the read model (queries, denormalized for speed), keeping the read side updated by consuming the write side’s events. Both are powerful and both buy you operational complexity and eventual consistency — reach for them when the audit trail or independent read/write scaling genuinely justifies it, not by default.
The outbox pattern: defeating the dual-write problem
Now the heart of it. The hook’s bug is the dual-write problem: a handler must update its database and publish an event, but those are two separate systems with no shared transaction. Whatever order you choose, a crash in the gap corrupts state: commit-then-publish can lose the event (the hook); publish-then-commit can emit an event for a write that then rolls back (a phantom event downstreams act on). You cannot make a DB commit and a broker publish atomic directly.
The transactional outbox turns two writes into one. In the same local database transaction that writes the business state, you also insert the event into an outbox table. Because both rows are in one transaction, they commit or roll back together — the atomicity you couldn’t get across systems, recovered by keeping both writes in the system that has transactions. A separate relay process then reads unpublished rows from the outbox and publishes them to the broker, marking each done once acknowledged.
ONE local transaction:
INSERT order (id=8842, ...) ← business state
INSERT outbox (event="OrderPlaced") ← event to publish
COMMIT ← both, or neither
Relay (separate, async):
SELECT * FROM outbox WHERE published = false
publish each to broker → on ack, mark published = trueThe relay publishes at-least-once (it may crash after publishing but before marking the row done, re-publishing on restart) — which loops us straight back to lesson 1: consumers must be idempotent, because the outbox guarantees the event is never lost, not that it’s delivered exactly once. The outbox trades “lost events” for “duplicate events,” and duplicates you already know how to handle.
Eventual consistency is a UX problem, not just a backend one
Asynchrony means the rest of the system learns about a fact after it happened — there is a window where the order exists but inventory hasn’t reacted yet. This eventual consistency is the unavoidable consequence of trading synchronous calls for events, and the senior mistake is treating it as purely a backend concern. It leaks straight into the UX: a user who places an order and immediately refreshes may not see it; a “your balance” view fed by a CQRS read model may lag the write by a beat. You must design the interface for the lag — optimistic UI that shows the expected result immediately, a “processing…” state, read-your-own-writes for the acting user, or explicit “may take a moment” messaging. Pretending the system is synchronous when it isn’t is how you ship confusing, “did my click work?” experiences.
▸Common mistake
A subtle outbox failure: doing the outbox insert but publishing the event directly from the handler “to save a hop,” instead of from the relay. Now you’re back to a dual write — the handler can crash after committing the outbox row but you’ve also tried an inline publish that may or may not have happened, and you’ve gained nothing. The discipline is strict: the handler’s only job is the local transaction (business row + outbox row); a separate relay owns all publishing. Keep them separate or the pattern silently degrades back into the bug it was meant to fix.
A handler does `db.commit(order)` then `broker.publish(event)`. Occasionally the order is saved but the event never fires. What is this, and how does the outbox fix it?
A 6-step order fulfilment process must support refunds/stock-release on failure, and ops keeps asking 'where is this order in the flow?' Choreography or orchestration?
The outbox pattern defeats the dual-write problem by writing the business row and the event row in one local database transaction, so the event is never lost — but because a separate relay then publishes it at-least-once, consumers must still be _______ to absorb the duplicates.
This is composition altitude — choosing coordination styles and the outbox for a design. The dedicated queues track goes deeper into broker-side transactional/idempotent producers and exactly-once stream processing; reach there when you’re implementing the relay against a specific broker, and stay here to decide choreography vs orchestration and whether you need the outbox at all.
- 01What inverts in event-driven vs request-driven, and what does it cost?
- 02Choreography vs orchestration — define each and the rule of thumb.
- 03What is the dual-write problem and how does the outbox solve it?
Event-driven architecture coordinates by emitting facts rather than making calls: the producer announces “what happened” and the system’s behaviour emerges from independent reactions, buying loose coupling at the cost of a traceable control flow and inheriting asynchrony’s consequences. Multi-step processes are coordinated by choreography (services react to each other’s events — decoupled, best for simple flows, but the process logic and state live nowhere) or orchestration (a central saga drives the steps and runs compensating actions on failure — best when the flow is complex or must not half-complete). Event sourcing (events as the source of truth, replayed to derive state) and CQRS (split write and read models) ride along on a retained log and are worth their complexity only when audit or independent scaling justifies it. The load-bearing pattern is the transactional outbox, which defeats the dual-write problem: write the business row and the event row in one local transaction, then let a separate relay publish asynchronously — recovering atomicity inside the one system that has transactions. Because the relay publishes at-least-once, the guarantee is “never lost, possibly duplicated,” so consumers must be idempotent; and the resulting eventual consistency is a UX problem you design for (optimistic UI, processing states, read-your-own-writes), not a backend detail you can hide. The dedicated queues track covers broker-side transactional producers; stay here to choose the coordination style and decide whether you need the outbox. Now when you see an order that exists in the database but never triggered shipping — check the gap: was there a dual write with no outbox, or a choreography flow with no one responsible for the missing step?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.