Eventual consistency as a design choice
Designing for eventual consistency means solving the dual-write problem with the transactional outbox, making consumers idempotent, and accepting that read-your-writes requires explicit design. Consistency is a structural decision, not an afterthought.
The billing platform’s OrderService saved order state to PostgreSQL and then published an OrderPlaced event to Kafka. Simple enough. But under load, the service occasionally crashed after the database write and before the Kafka publish. The event was lost. Downstream services — inventory, payment, shipping — never received it. Orders were stuck in a limbo state: confirmed in the database, invisible to everyone else. The team added retry logic. Now, on crash-then-restart, the service sometimes published the event twice. Downstream services processed the same order twice, double-charging customers. The dual-write problem — two separate I/O operations with no atomicity — had two failure modes, and fixing one exposed the other.
The dual-write problem
Any time a service must update its own state AND notify the rest of the system, it faces two separate I/O operations:
- Write to the database (the service’s source of truth)
- Publish to a message broker (the signal to downstream services)
These two writes have no atomicity guarantee. Any of four outcomes is possible:
| DB write | Broker publish | Effect |
|---|---|---|
| success | success | correct |
| success | failure | event lost — downstream stuck |
| failure | success | phantom event — downstream acts on data that was never persisted |
| failure | failure | clean failure — no harm, retry safe |
The third case (phantom event) is particularly dangerous: downstream services process an event for an order that does not exist in the source database. In billing, this means charging a customer for a non-existent order.
The first two failure cases — lost events and phantom events — cannot be eliminated by adding retry logic. Retry logic that republishes after a crash fixes “event lost on crash” but introduces “event published twice on retry.” The dual-write problem is structural, not operational.
A developer proposes solving the dual-write problem by wrapping the DB write and Kafka publish in a try/catch: if Kafka publish fails, roll back the DB transaction. A senior engineer says this does not solve the problem. Why?
The transactional outbox
The transactional outbox solves the dual-write problem by removing the second write from the service’s request path. Instead of publishing directly to Kafka, the service writes the event to an outbox table in the same database transaction as the state update. The DB transaction is the single source of truth.
BEGIN;
INSERT INTO orders (id, customer_id, status) VALUES ($1, $2, 'pending');
INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'order', $1, 'OrderPlaced', $3);
COMMIT;If the DB transaction commits, both the order row and the outbox row are durably written. If it fails, neither is written. Atomicity is guaranteed by the database.
A separate outbox relay process reads uncommitted outbox rows and publishes them to Kafka. The relay marks rows as processed after successful publish. The relay can be:
- Polling-based: a background thread or service that queries
SELECT * FROM outbox WHERE published_at IS NULL LIMIT 100on an interval - CDC-based (change-data-capture): tools like Debezium read the database’s WAL (write-ahead log) and stream outbox insertions directly to Kafka without polling
▸lesson.inset.note
The outbox pattern shifts the reliability problem to the relay. The relay guarantees at-least-once delivery: it reads an outbox row, publishes to Kafka, and marks the row as processed. If the relay crashes after publishing but before marking, it will republish the same event on restart. This is intentional — it is the source of the at-least-once guarantee. Idempotent consumers handle the duplicate. This is why idempotency is structural, not optional: the outbox relay’s correctness depends on consumers tolerating duplicates.
The billing platform implements the transactional outbox. A developer says: 'Now that we have the outbox guaranteeing exactly-once delivery, our consumers do not need to handle duplicates.' What is wrong with this claim?
Idempotent consumers
At-least-once delivery is not a delivery failure — it is a guarantee. Every consumer in an event-driven system must be designed to process the same event multiple times and produce the same result as processing it once.
The standard structural approach: maintain a deduplication table that records processed event IDs. Before processing an event, check if its ID is already in the table. If yes, skip. If no, process and insert.
BEGIN;
-- Check for duplicate
SELECT 1 FROM processed_events WHERE event_id = $1;
-- If no row: process the event and record it
INSERT INTO processed_events (event_id, processed_at) VALUES ($1, NOW());
UPDATE inventory SET reserved = reserved + $2 WHERE product_id = $3;
COMMIT;The deduplication check and the business update must be in the same transaction. If they are separate, a crash between the business update and the deduplication record insert produces a processed event with no deduplication record — the next delivery will process it again.
Read-your-writes under eventual consistency
After a user submits an order, they expect to see it immediately in their dashboard. If the dashboard reads from an async projection fed by events, the order may not appear for hundreds of milliseconds. The user submitted, received a success response, and the UI shows nothing.
Solutions:
- Read-after-write from the write side: for the user’s own recent writes, read directly from the service that owns the state (the write model), bypassing the async projection. This requires the UI to know which service to query for the user’s own recent data.
- Optimistic UI update: update the UI immediately on submit, before the event is processed downstream. If the server rejects the submission, roll back the UI. This does not fix the underlying eventual consistency but hides its latency from the user.
- Sticky sessions to the same replica: for read replicas of the write model, route the user’s reads after a write to the same replica that received the write, until replication catches up.
A team builds a payment dashboard that reads from a read model updated by Kafka events. A product manager reports: 'After a payment is confirmed, users sometimes do not see it in their dashboard for 2-3 seconds.' The team's lead says this is 'working as designed.' A developer wants to fix it. Which fix is architecturally correct?
- 01What are the two dangerous failure modes of the naive dual-write pattern (DB write then broker publish)?
- 02Describe the transactional outbox in one paragraph: what is written where, and who is responsible for delivery?
- 03Why must the deduplication check and the business update in an idempotent consumer be in the same database transaction?
The billing platform’s dual-write bug had two modes: lost events when the service crashed after the DB write, and duplicate events when retry logic republished on restart. Neither mode was fixable by adjusting retry parameters — the problem was structural.
The transactional outbox removes the second write from the request path entirely. The event is written as an outbox row in the same database transaction as the state update. The relay handles delivery asynchronously. Atomicity is guaranteed by the database; eventual delivery is guaranteed by the relay’s retry behavior.
The relay’s retry behavior is the source of at-least-once delivery — and at-least-once means consumers will receive duplicates. Idempotency is not a nice-to-have; it is the contract that makes the outbox pattern work. A consumer that processes the same event twice and produces a double-charge is not idempotent. The deduplication table, transactionally tied to the business update, is the structural fix.
Eventual consistency under this design is a deliberate choice: the system accepts that downstream state lags the source by milliseconds to seconds, in exchange for availability, throughput, and the freedom to evolve services independently. Read-your-writes is the most common user-visible symptom of that lag, and it requires its own structural answer — not a sleep and a prayer.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.