Design ad-click aggregation
Design a system that counts ad clicks for billing: stream processing with Kafka and Flink, exactly-once via idempotency, windowed aggregation, late events and watermarks, deduplication, and a batch reconciliation layer that corrects the fast path.
An advertiser opened a dispute: the platform’s dashboard said their campaign got 1.2 million clicks last month, but their own landing-page analytics counted 1.0 million. Two hundred thousand clicks — at a few cents each — were being billed for nothing. The engineering team dug in and found three culprits braided together: a network retry that sent some click events twice, a stream processor that double-counted after a crash-and-replay, and a batch of clicks that arrived hours late from users on flaky mobile connections and got dropped from the window they belonged to. None of these is exotic. They are the default behaviour of a naive streaming pipeline, and they are exactly why ad-click aggregation — counting events for money — is one of the hardest “just count things” problems in the field.
Requirements
The system ingests a firehose of ad-click events and produces aggregated counts — clicks per ad, per minute, per region — that feed real-time dashboards and the billing system. Because the output is money, the bar is far higher than for analytics.
Functional. Accept click events (ad_id, user_id, timestamp, click_id, region). Aggregate them into time windows (per-minute, per-hour counts). Serve fast queries (“clicks in the last 5 minutes”) for the advertiser dashboard, and produce an authoritative daily total for billing. Filter fraud and invalid clicks. Let advertisers slice by dimensions (ad, campaign, geo).
Non-functional. Correctness is paramount — every billable click counted exactly once, no more, no less, because over-counting overbills (disputes, refunds, legal risk) and under-counting loses revenue. High throughput: a large network sees hundreds of thousands of clicks per second at peak. Low latency for the dashboard (seconds-fresh), but billing can tolerate hours of latency in exchange for being right. The system must survive failures without corrupting counts.
The tension that shapes everything: a click is an event in time, the network delivers events out of order and sometimes twice, and processors crash mid-flight. Getting an exact count out of that mess is the whole job.
Estimation
peak clicks = 1,000,000 clicks/sec (large ad network at peak)
event size = ~100 bytes per click event
peak ingest = 10^6 × 100 bytes = 100 MB/sec into the log
clicks per day = ~1,000,000 × 86,400 (avg ~1/3 of peak) ≈ 3 × 10^10/day
raw retention = 3 × 10^10 × 100 B ≈ 3 TB/day raw events
aggregated output = ad_count × windows ≪ raw (counts are tiny vs events)Two observations drive the design. First, raw event volume is huge (100 MB/s, multi-TB/day) but aggregated output is tiny — the system’s job is to collapse a firehose into small counts, so the expensive part is the streaming compute, not the result storage. Second, you must keep the raw events (in the log / cold storage) even after aggregating, because that raw record is what the batch reconciliation layer re-reads to correct the fast path and what you show an advertiser in a dispute. The aggregates are derived; the events are the source of truth.
High-level design
This is the lambda architecture in its classic form: a speed layer (streaming) for low-latency-but-approximate dashboard counts, and a batch layer that re-reads the immutable raw events to compute correct totals and reconcile against the speed layer. The durable, partitioned log (Kafka) is the linchpin — it’s the replayable source of truth that both layers consume, and replayability is what lets the batch layer recompute exactly and the stream layer recover after a crash. A modern alternative, kappa, drops the separate batch code and runs everything through the stream engine, reprocessing by replaying the log when correction is needed.
Deep dive
Windowing and watermarks: handling event time
A click happens at a moment — its event time, stamped on the device. But it arrives at the server later — processing time — and the gap is wildly variable: a user on subway Wi-Fi might emit a click whose event arrives 40 minutes later. If you aggregate by processing time (“count whatever showed up this minute”), late clicks land in the wrong minute and your per-minute billing is wrong. So you aggregate by event time, grouping clicks into windows by when they happened, not when they arrived.
That raises the closing problem: a window for 14:05:00–14:05:59 can’t stay open forever waiting for stragglers, but closing it instantly drops the late ones. The mechanism is a watermark — the engine’s assertion “I believe I have now seen all events up to event-time T.” When the watermark passes a window’s end, the window closes and emits its count. The watermark is deliberately lagged behind the latest event time (say by a few minutes) to absorb normal lateness, trading latency for completeness.
window [14:05:00, 14:06:00)
click A event-time 14:05:10 arrives 14:05:12 ✅ in window
click B event-time 14:05:40 arrives 14:05:43 ✅ in window
watermark reaches 14:06:00+lag → window CLOSES, emits count = 2
click C event-time 14:05:55 arrives 14:09:30 ⚠️ LATE — window already closed▸Why this works
Why not just wait longer before closing every window, so nothing is ever late? Because the watermark lag is a direct latency tax on every result, paid to catch a rare straggler. If 99.9% of clicks arrive within 10 seconds but 0.1% take 30 minutes, setting the watermark lag to 30 minutes makes every dashboard number 30 minutes stale to rescue one click in a thousand. The senior move is to size the watermark for the common case (say a minute or two), accept that truly late events miss their window, and handle them with a separate mechanism — an allowed-lateness side output or, better, the batch reconciliation layer that re-reads everything with no time pressure. You don’t make the fast path slow to be correct; you make the fast path fast and let a slow path fix it.
Exactly-once, idempotency, and dedup
“Exactly-once” is the headline guarantee, and it’s the subtlest. The network and the pipeline both create duplicates: a client times out and retries the same click (so the event arrives twice with the same click_id); a stream processor crashes after counting an event but before committing its offset, then replays it. Naively, both inflate the count — the hook’s over-billing.
The fix has two halves, for the two sources of duplication:
- Source dedup by idempotency key. Every click carries a unique
click_idgenerated at the edge. The aggregator dedups on it — within a window, aclick_idalready seen is dropped. This kills client/network retries. (Dedup state is itself bounded by the window: you only need to remember IDs for the window still open.) - End-to-end exactly-once processing. This is not “the event is delivered once” (impossible over an unreliable network); it’s “the effect on the result is applied once.” Modern stream engines achieve it by binding the output commit and the input offset advance into one atomic, replayable unit — Flink does this with periodic checkpoints (a consistent snapshot of all operator state plus the consumed offsets), and a transactional sink (Kafka transactions, or an idempotent write keyed by window) so that on recovery the engine rewinds to the last checkpoint and re-emits without double-applying.
The mental model: exactly-once is at-least-once delivery plus idempotent/transactional application. You can’t stop duplicates arriving; you make duplicate application impossible.
▸Common mistake
The classic mistake is believing a vendor’s “exactly-once” flag means messages are physically delivered exactly once — and then putting a non-idempotent side effect (charge a card, send an email, increment a raw external counter) directly in the stream operator. When the job recovers from a checkpoint and replays, that side effect fires again, because exactly-once only covers state inside the engine’s checkpoint/transaction boundary. The card charge is outside it. The fix is to keep external side effects idempotent in their own right (idempotency keys on the charge) or to push them past the boundary into a transactional sink the engine controls. Treat “exactly-once” as a property of the engine’s internal state and its committed output, never as a promise about arbitrary effects you trigger mid-stream.
Reconciliation: the batch layer audits the stream
The streaming layer is fast but approximate: it may have closed a window before a late click arrived, or briefly double-counted across a crash recovery edge. So a batch reconciliation job runs (hourly/daily) over the complete set of raw events in cold storage — no watermark pressure, every late click now present — and computes the authoritative total. It compares against the streaming total and corrects the billing figure. This is the safety net the hook’s team was missing: the dashboard can be a little wrong and self-heal within minutes, but the money number comes from a batch job that has seen everything. The pattern generalizes: let the fast path be eventually-consistent and the slow path be the source of truth for anything billable.
Bottlenecks & tradeoffs
The core tradeoff is latency vs completeness, embodied in the watermark: a longer lag catches more late events but staleness rises, so you tune it for the common case and let batch handle the tail. Hot partitions bite at the log layer — if you partition by ad_id and one viral ad gets 30% of clicks, that partition’s consumer is overwhelmed; you add a salt/sub-key to spread a hot key across partitions, then re-aggregate. State size is the stream engine’s constraint: per-window dedup sets and aggregation state must fit in managed state (RocksDB-backed, checkpointed) — bound it by closing windows promptly and expiring dedup keys. Lambda’s cost is maintaining two codebases (stream + batch) that must agree, which is exactly why kappa (one engine, replay for correction) is attractive — at the price of a heavier streaming system and long replay times. And underneath it all, exactly-once has a throughput cost: checkpoints and transactions add coordination, so a system that can tolerate rare double-counts (pure analytics) often runs at-least-once for speed — but billing cannot, so here you pay the tax.
A click event arrives 30 minutes after it happened because the user was on a flaky connection. Your per-minute windows aggregate by event time with a 2-minute watermark lag. What happens to this click, and what's the correct way to still bill it?
Your stream job is configured 'exactly-once,' yet you put a `chargeAdvertiser()` HTTP call directly inside the aggregation operator. After a crash, the job recovers from its last checkpoint and replays — and some advertisers get charged twice. Why?
Practical exactly-once is really at-least-once delivery plus _______ application: you cannot stop duplicate events from arriving over an unreliable network, so instead you make applying the same event twice have no extra effect — via a dedup key or a transactional commit bound to the input offset.
- 01Why aggregate by event time, and what role does the watermark play?
- 02What does 'exactly-once' actually mean here, and how is it achieved?
- 03What is the batch reconciliation layer for, and how does it relate to lambda/kappa?
Counting ad clicks for billing is a correctness problem disguised as a counting problem, because the network delivers events out of order, late, and sometimes twice, and processors crash mid-flight — the hook’s braided over-billing. The architecture is lambda over a durable, replayable log: a streaming speed layer gives seconds-fresh dashboard counts, and a batch layer re-reads the immutable raw events to compute the authoritative billing total and reconcile away the fast path’s small errors (kappa is the one-engine, replay-to-correct alternative). The fast path aggregates by event time so late clicks don’t land in the wrong minute, closing windows on a watermark whose lag is sized for the common case — accepting that rare stragglers miss the window and are recovered by batch. Exactly-once is engineered as at-least-once delivery plus idempotent/transactional application: dedup retries by click_id, bind output commit to input offset via checkpoints and a transactional sink — but it only covers state inside the engine, so external side effects must be made idempotent themselves. The standing tradeoffs are latency vs completeness (the watermark), hot partitions on viral ads (salt the key), bounded stream state (close windows, expire dedup keys), and the exactly-once throughput tax you pay because, unlike analytics, billing cannot tolerate a single double-count. Now when you see a stream job wired to an external charge or email in an operator, you know exactly why it will double-fire after a crash recovery — and what to do: push that side effect past the transactional boundary or make it idempotent in its own right.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.