infra · advanced · 6d
At-least-once job queue
Build a durable job queue on Postgres with visibility timeouts and idempotent consumers, so a crashed worker never drops a job.
Deliverable
A queue where every enqueued job runs at least once even across worker crashes, with duplicates made safe by idempotent handlers.
Milestones
0/3 · 0%- 01Claim jobs without double-grab
Claim jobs with FOR UPDATE SKIP LOCKED so two workers never grab the same row.
Definition of done- Two concurrent workers running the claim query never receive the same job row; a skipped-locked row is picked up by the other worker.
- The claim marks the row in-flight in the same transaction that selects it.
- 02Re-queue jobs from dead workers
Add a visibility timeout that re-queues a job whose worker died before acking.
Definition of done- A job whose worker dies before ack becomes claimable again after the visibility timeout, not lost.
- A still-running worker extends its lease before the timeout so its job is not stolen mid-flight.
- 03Make re-delivery a no-op
Make a consumer idempotent via an idempotency key so a re-delivered job is a no-op.
Definition of done- Processing the same job twice (same idempotency key) produces exactly one effect; the second run is a no-op.
- A poison job that always fails lands in a dead-letter store after N attempts instead of looping forever.
Starter
- README.md
- src/queue.ts
- test/queue.test.ts
Unzip, implement the stubs, then run the tests until they pass: bun test
Rubric
| Junior | Mid | Senior | |
|---|---|---|---|
| Claim atomicity | A worker selects a pending job and updates it in two separate statements; under concurrent load, two workers occasionally grab the same row. | Claim is a single UPDATE ... WHERE state='pending' RETURNING with FOR UPDATE SKIP LOCKED — concurrent workers never double-grab, and a skipped row is immediately available to the next poller. | You can reason about the contention model: SKIP LOCKED scales to many workers with no lock-wait latency, but concentrates all pending work on the oldest rows; you measure the claim throughput ceiling and know when to partition the queue table. |
| Visibility timeout & re-delivery | A crashed worker's job is stuck in 'claimed' until manually intervened; there is no automatic re-queue. | A sweeper or lease check re-queues jobs whose visibility timeout expired; a live worker extends its lease via heartbeat so long jobs are not stolen. | You set the timeout against p99 job duration and can articulate the two failure modes: too short means a slow-but-alive worker races its own task; too long means a real crash leaves the lane stalled for minutes — you document the chosen value and its rationale. |
| Idempotent consumer & dead-letter | Re-delivered jobs are processed again, occasionally producing duplicate side effects; no attempt cap exists. | A dedup key recorded in the same transaction as the effect makes re-delivery a no-op; after N failures the job moves to a dead-letter table instead of looping. | You reason about the atomicity boundary: if the effect and dedup key are in separate commits, a crash between them either double-applies or replays forever — your design makes both impossible, and you prove it with a chaos test killing workers at every crash point. |
Reference walkthrough (spoiler)
Why at-least-once is the honest baseline: exactly-once delivery requires distributed coordination that is either very expensive (two-phase commit) or impossible across heterogeneous systems. Any durable queue built on a single datastore can only promise at-least-once — a job runs again if the worker crashes before acking — and correctness is pushed to the consumer via idempotency.
FOR UPDATE SKIP LOCKED as the claim primitive: it combines select and lock in one statement so no second worker can observe the same row, and the SKIP avoids lock-wait pile-ups — a blocking FOR UPDATE would serialize all workers on a hot table instead of letting them fan out.
The visibility timeout tuning trap: the right timeout is slightly above p99 job duration — not p50, not a conservative 10x guess. Too short and you race live workers; too long and crashed-worker jobs stall the lane. A heartbeat that renews the lease on a sub-timeout interval is the correct fix for long jobs, not a larger timeout.
Dead-letter queue depth as the earliest signal: a poison job crashes every worker that touches it, and without a DLQ it occupies the queue head forever, blocking every other job behind it. DLQ depth rising is the first observable symptom that something upstream is broken or a payload is malformed — alert on it, not on job latency alone.
Make it senior
- Add exponential backoff with jitter and a dead-letter queue after N failures.
- Run a chaos test killing workers mid-job; assert zero lost and zero unsafe duplicate effects.