Event sourcing: the append-only log as source of truth

A customer disputes a charge: “I never set my plan to annual.” Support pulls the row — plan: annual — and shrugs. But the accounts table only holds the current value; the UPDATE that set it overwrote whatever was there before, and the application logs rotated out two weeks ago. Nobody can say what the plan was on the 14th, who changed it, or whether a buggy webhook did it. In an event-sourced system this is a thirty-second query: replay the stream up to that timestamp and read the answer. The difference is not a better log — it’s that the log is the database.

State is a left-fold over events

A normal CRUD table stores the latest snapshot and destroys history on every UPDATE. Event sourcing inverts this: the append-only event store holds an ordered, immutable sequence of facts — AccountOpened, PlanChanged, CardDeclined — and the current state is just what you get by folding a reducer over them from the beginning. state = events.reduce(apply, initial). The events are the source of truth; the “current state” is a derived cache you can throw away and recompute at any time.

That single inversion is where every benefit and every cost comes from. You get a complete audit trail for free (the log is the audit). You get temporal queries — state as-of any past instant, by folding only the events before that timestamp. You get debugging by replay: copy the production stream into staging, run it through the new code, and watch the exact bug reproduce deterministically. None of this is bolted on; it falls out of never throwing data away.

The append-only constraint is load-bearing. You never UPDATE or DELETE an event. A mistake is corrected by appending a compensating event (PlanCorrected), the same way an accountant never erases a ledger entry — they post a reversing one. This is why event stores optimize for one thing: fast appends and fast sequential reads of a stream.

Event sourcing is not “Kafka” and not a change log

This is the distinction seniors get wrong most often. A change-data-capture log or a plain Kafka topic records events, but that alone is not event sourcing. The defining property is that the log is the authoritative source of truth and state is rebuilt from it — not a side-channel of changes emitted after a database already committed the truth elsewhere.

Kafka can serve as an event store, but with sharp caveats. Log compaction — Kafka’s headline feature — keeps only the latest value per key, which directly destroys the history event sourcing depends on; you must use retention-based topics with log.retention.ms = -1 (infinite), not compacted ones. Per-aggregate optimistic concurrency (“append only if I’m at version N”) has no clean primitive in Kafka, whereas a purpose-built store like EventStoreDB makes expectedVersion a first-class append condition. The common production pattern: keep the raw event topic forever as the source, and publish derived current-state to a separate compacted topic for read models — the compacted topic is a projection, never the truth.

CQRS: projections are disposable read models

You cannot serve a query like “show the dashboard” by folding the entire log on every request — that would be ruinously slow. So event sourcing almost always pairs with CQRS: the write side appends events; the read side runs projections that consume the stream and materialize purpose-built read models (a SQL table, an Elasticsearch index, a denormalized cache). Each projection is a tiny program: for every event, update its own table. Because projections are derived, they are disposable — drop the table, replay the log, get it back. Need a brand-new view six months in? Write a projection and replay history through it; the data was always there.

Two properties make this safe in production. First, projections must be idempotent: the same event may be delivered more than once (retries, at-least-once delivery), so applying it twice must equal applying it once. The standard mechanism is to track the last processed event version per stream and ignore any event whose version is <= the last seen. Second, the read side is eventually consistent: there is real lag between an event being appended and the projection catching up. EventStoreDB and similar stores document this read-model lag explicitly — the read model “converges to the correct state given time,” it is not guaranteed current at any instant.

The hard parts: versioning, GDPR, and replay cost

This is where event sourcing earns its reputation. Schema versioning is unavoidable and permanent: you can never delete an old event shape, because old events written in that shape live in the log forever and must still be replayable. When OrderPlaced gains a field, every historical OrderPlaced lacks it. The standard answer is upcasting — a chain of transformation functions that lift an old event shape to the current shape at deserialization time, so the rest of your code only ever sees the latest version. Upcasters accrete; they are code you carry indefinitely.

GDPR’s “right to be forgotten” collides head-on with an immutable log. You legally must erase a user’s personal data, but the log is append-only and you cannot rewrite history without breaking every downstream replay and audit guarantee. The dominant technique is crypto-shredding: encrypt each user’s PII with a per-user key stored outside the log; to “forget” them, throw away the key, rendering the ciphertext permanently unrecoverable while the event structure stays intact. Note the sharp caveat seniors must flag: regulators may still treat undeletable encrypted PII as personal data, so crypto-shredding is a pragmatic mitigation, not a guaranteed legal slam-dunk — get counsel involved.

Finally, replay cost is unbounded if you do nothing. Folding millions of events to load one aggregate gets slow; a financial system replaying terabytes of price ticks can take minutes per rebuild. The fix is snapshotting: periodically persist the folded state at version N, then on load read the snapshot and replay only events after N. Snapshots are a pure performance optimization and a footgun — if a snapshot drifts from the events (logic changed, a write got lost mid-stream), it silently serves wrong state and masks the bug because replay never re-derives it. Defensive teams checksum snapshots and rebuild on mismatch.