Sagas, compensations and the outbox pattern

Why not one transaction: 2PC is a fragile SPOF

A “place order” operation touches three services — order, inventory, payment — and each owns its own database. The instinct from monolith days is to wrap all three writes in one transaction so they commit or roll back together. You can’t. An ACID transaction lives inside a single database; it cannot span three independent databases owned by three independent services.

The classic answer to “atomic across multiple databases” is two-phase commit (2PC / XA): a transaction coordinator asks every participant to prepare, and once all vote yes, tells them all to commit. It does give you atomicity — and almost every modern microservice stack refuses to use it, for three reasons that compound under load:

• The coordinator is a single point of failure. If it dies after participants have voted to prepare but before it broadcasts commit, every participant is stuck holding locks, waiting for a decision that may never come.
• Locks are held across the network. Between prepare and commit, each database holds its locks for the full round-trip to every other participant. Under partition or slowness, those locks sit there, and throughput collapses.
• Availability tanks under partition. 2PC is a CP protocol in the brutal sense: if any participant or the coordinator is unreachable, the whole transaction blocks. One slow service stalls all of them.

So the consistency model changes. Instead of one transaction that is atomic now, you accept eventual consistency: the steps happen one at a time, each atomic on its own, and the system converges to a correct state shortly after — not instantly. The pattern that does this is the saga.

The saga: local transactions plus compensations

A saga is a sequence of local transactions, each one running inside a single service’s own database, linked by events or commands. Reserve inventory is one local transaction in the inventory DB. Charge payment is another local transaction in the payment DB. Each commits on its own. The saga is the chain.

The hard part is failure. If step 3 (charge payment) fails, you can’t roll back steps 1 and 2 — they already committed. A committed local transaction is gone; there is no ROLLBACK to issue. Instead the saga runs compensating transactions: a new local transaction that semantically undoes the prior one. You don’t un-charge a payment; you issue a refund. You don’t un-reserve inventory; you release the reservation.

// A saga step is a local txn + a named compensation. The orchestrator runs the
// forward action; if a LATER step fails, it runs each completed step's compensate().
interface SagaStep {
  action: () => Promise<void>;      // local transaction in ONE service
  compensate: () => Promise<void>;  // semantic undo, runs AFTER commit, may itself fail
}

const placeOrderSaga: SagaStep[] = [
  { action: () => inventory.reserve(orderId, items),
    compensate: () => inventory.release(orderId) },        // not a rollback — a release
  { action: () => payment.charge(orderId, total),
    compensate: () => payment.refund(orderId) },           // not a rollback — a refund
];

There are two ways to coordinate the chain, and choosing between them is a senior decision:

• Choreography — no central coordinator. Each service listens for events and emits the next one. Order service emits order.created; inventory reacts, reserves, emits inventory.reserved; payment reacts to that. Decentralized and simple for short flows — but the overall flow exists nowhere in code. To answer “what happens when an order is placed” you read five services’ event handlers and reconstruct it in your head. It rots as the flow grows.
• Orchestration — a central saga orchestrator owns the flow. It sends explicit commands (reserveInventory, then chargePayment), tracks the saga’s state in its own table, and decides when to fire compensations. It’s another component to build and run, but the flow is in one place: explicit, observable, testable, restartable.

The senior rule of thumb: choreography for short 2–3 step flows; orchestration when the flow is long, branchy, or needs visibility — anything you’ll have to debug at 3am wants a state machine you can query.

Together these two styles cover the coordination spectrum: choreography gives you zero new components and zero centralized state, while orchestration gives you one place to answer “where is this saga right now?” Without that single place, a failed payment in a five-service choreography requires you to read five event logs to reconstruct what happened.

The dual-write problem and the transactional outbox

The Hook’s incident is the dual-write problem, and it’s the real reason this lesson exists. A service almost always has to do two writes as part of one logical step: update its own database (the order row) and publish an event so other services learn about it (order.created). Those are two systems — the DB and the broker — with no shared transaction. Whatever order you do them in, a crash in the gap breaks you:

• DB first, then publish, crash in between → the order exists but no event was sent. Downstream services never learn the order exists. Inventory never reserves; the customer waits forever. This is the Hook.
• Publish first, then write DB, DB fails → you published order.created for an order that doesn’t exist. Now downstream services act on a lie, and you have a phantom order rippling through the system.

The transactional outbox fixes this by removing the second system from the critical write. In the same local database transaction that writes the business row, you also INSERT the event into an outbox table. Both rows commit together or neither does — it’s one transaction in one database, so there is no gap:

// Both writes in ONE local transaction: business row + event row commit atomically.
await this.dataSource.transaction(async (manager) => {
  const order = await manager.getRepository(Order).save(newOrder);
  await manager.getRepository(OutboxEvent).insert({
    aggregateId: order.id,
    type: 'order.created',
    payload: JSON.stringify({ orderId: order.id, items: order.items }),
    // status defaults to 'pending'; a relay flips it to 'sent' after publishing
  });
}); // <- if the pod dies here, NEITHER row committed. No half state. No lost event.

A separate message relay then carries outbox rows to the broker. Two common implementations:

• Polling publisher — a background worker SELECTs pending rows, publishes each to the broker, and marks it sent (or deletes it). Simple, no extra infrastructure; the trade-off is poll latency and load.
• Change Data Capture (CDC) — a tool like Debezium tails the database’s write-ahead log, sees the outbox INSERT, and publishes it. Near-real-time and no polling, but it’s a serious piece of infrastructure to operate.

Either way the relay guarantees at-least-once publication: it keeps trying until the broker acknowledges, so a publish failure or relay crash means a retry, never a lost event. At-least-once means a consumer can see the same event twice — which is exactly where the idempotent consumers from the previous lesson (L03) earn their keep: dedupe on the event id downstream, and at-least-once delivery plus idempotent processing gives you effectively-once. The outbox doesn’t replace delivery semantics; it feeds them a reliable source.

// The relay: at-least-once publish. Keeps the event until the broker acks.
async function relayTick(manager: EntityManager, client: ClientProxy) {
  const pending = await manager.getRepository(OutboxEvent)
    .find({ where: { status: 'pending' }, take: 100, order: { id: 'ASC' } });
  for (const evt of pending) {
    await firstValueFrom(client.emit(evt.type, JSON.parse(evt.payload))); // may retry
    await manager.getRepository(OutboxEvent).update(evt.id, { status: 'sent' });
  }
}