Sagas
A saga coordinates a business process across services using local transactions and compensating actions — not a distributed 2PC lock. Compensation is not rollback: it appends a corrective action. Sagas accept visible intermediate states.
The B2B billing platform needed to handle order placement as a cross-service operation: reserve inventory, charge payment, update order status — all or nothing. The team tried a distributed transaction using two-phase commit. It worked in staging. In production, the payment gateway sometimes took eight seconds to respond. The distributed transaction held inventory records locked for those eight seconds. Under load, lock timeouts cascaded: inventory locks starved other orders, timeouts triggered retries, retries piled on more locks. The system ground to a halt. They abandoned 2PC and implemented a saga instead — a sequence of local transactions, each committed independently, with explicit compensating actions to undo earlier steps if a later step failed.
Why 2PC fails in distributed systems
Two-phase commit (2PC) provides ACID atomicity across multiple participants: either all commit or all abort. The coordinator asks all participants to prepare (phase 1), waits for acknowledgment from all, then sends commit or abort (phase 2). If any participant says “no” in phase 1, the coordinator aborts all.
This sounds correct. The problem is operational:
Long-held locks. In phase 1, each participant holds its local resources locked until it receives the phase 2 commit or abort message. If the payment gateway takes 8 seconds, the inventory lock is held for 8 seconds. Under concurrency, slow participants block unrelated transactions.
Coordinator failure. If the coordinator crashes between phase 1 and phase 2, participants are stuck in an “uncertain” state — they prepared but do not know whether to commit or abort. They cannot release their locks until the coordinator recovers. Recovery can take minutes.
Participant unavailability. 2PC requires all participants to be available simultaneously. If the shipping service is down during phase 1, the entire transaction aborts — even if payment and inventory are fully ready.
The failure mode is not “sometimes the transaction fails” — it is “sometimes the entire system stalls with held locks and uncertain participants.” For high-throughput B2B billing, this is not acceptable.
▸Why this works
Sagas replace coordination with compensation. Instead of requiring all participants to be available simultaneously under a shared lock, each participant commits its local transaction independently as soon as it succeeds. Compensation handles failures after the fact, by appending corrective actions. This shifts the failure model from “stall” to “compensate” — which is recoverable and non-blocking.
What a saga is
A saga is a sequence of local transactions — each committed within a single service — that together implement a business process spanning multiple services. Each local transaction updates one service’s state and publishes an event or sends a message to trigger the next step.
Each step has a corresponding compensating action: a business operation that semantically undoes the prior step if a later step in the saga fails. Compensation is not a rollback — the prior transaction is already committed to the database. Compensation appends a new corrective event. If inventory was reserved, the compensation is to release that reservation — a new write, not an undo.
The key property sagas do NOT provide is isolation: intermediate states are visible to other transactions. If a saga reserves inventory at step 2 and then fails at step 3 (payment declined), between step 2 and the compensation, the inventory appears reserved to other readers. This is a structural consequence of using local transactions without a distributed lock.
Countermeasures for isolation loss:
- Semantic locks: mark resources as “pending saga” so other transactions treat them as reserved
- Commutative updates: design updates so they produce the same final state regardless of order, making intermediate state observability harmless
A team migrates from 2PC to sagas for their order fulfillment flow. A developer says: 'Sagas give us the same all-or-nothing guarantee as 2PC, just implemented differently.' A senior engineer corrects this claim. What is the precise difference?
The B2B order saga: steps and compensating actions
The billing platform’s order placement saga:
- OrderCreated —
OrderServicecreates the order record locally, status:pending. Compensation: cancel the order (mark ascancelled). - ReserveCredit —
PaymentServicereserves the customer’s credit limit for the order amount. Compensation: release the credit reservation. - ReserveInventory —
InventoryServicereserves the required stock. Compensation: release the inventory reservation. - ScheduleShipment —
ShippingServiceschedules the delivery. Compensation: cancel the scheduled shipment.
If ScheduleShipment fails (e.g., no carrier available for the address), the saga runs compensation in reverse: cancel the scheduled shipment (already failed), release the inventory reservation, release the credit reservation, cancel the order.
The billing platform's saga fails at the ScheduleShipment step after successfully committing ReserveCredit and ReserveInventory. The team's compensation implementation marks the saga as 'failed' in the saga log but does not run any compensating actions, reasoning that the inventory and credit reservations will expire automatically after 24 hours via a scheduled job. What is wrong with this approach?
Orchestrated vs choreographed sagas
The saga pattern can be implemented in either coordination style from lesson 01:
Orchestrated saga: a dedicated saga orchestrator service holds the complete saga state machine. It directs each participant step by step, tracks which steps have completed, and coordinates compensation when a step fails. The flow is explicit and visible. The orchestrator can emit saga-state events for observability. The downside is the coupling hub: all participants depend on the orchestrator.
Choreographed saga: each participant reacts to the previous step’s event and emits its own event on success or failure. The saga state is distributed across the events in the event log. No central coordinator. Participants are loosely coupled. The downside is that tracking the saga’s overall state — and knowing whether compensation has completed — requires correlating events across all participants with no single authoritative record.
For the B2B billing platform’s order saga with four to twelve services and frequent business-rule changes, an orchestrated saga is the better fit: the flow complexity warrants explicit coordination, and the business needs to be able to add rules like “pause if credit limit exceeded” in one place.
A new engineer joins the billing team and asks: 'Why can't we just wrap the order placement in a database transaction across OrderService, PaymentService, and InventoryService? We already use PostgreSQL everywhere.' What is the structural answer?
- 01What is a compensating action in a saga, and why is it not the same as a database rollback?
- 02Why do sagas not provide isolation, and what are the two main countermeasures?
- 03What is the main operational advantage of a choreographed saga over an orchestrated saga, and what is its main disadvantage?
The billing platform’s 2PC experiment failed not because the implementation was wrong, but because 2PC is structurally unfit for high-throughput distributed systems: slow participants hold locks that cascade into timeouts, and coordinator crashes leave participants in an uncertain state.
Sagas replace the distributed lock with local transactions and compensating actions. Each step commits independently. If a later step fails, compensation runs backwards — not as a rollback, but as a new corrective action appended to the record. The inventory that was reserved is explicitly released; the credit that was charged is explicitly refunded.
The cost is isolation: between the commit of step N and the execution of its compensation, the intermediate state is visible. This is not a bug — it is a structural consequence of replacing a distributed lock with local transactions. Countermeasures like semantic locks and commutative updates manage the exposure, but they do not eliminate it.
Saga coordination can be orchestrated (explicit, visible, centralised) or choreographed (implicit, decoupled, distributed state). The right choice depends on flow complexity, how frequently business rules change, and how important it is to have a single authoritative view of saga state at any given moment.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.