Message patterns and delivery semantics

Two patterns: a question vs a fact

Why does the pattern choice matter before you even pick a broker? Because the wrong one silently couples services in time — and that coupling is what turns a single downstream outage into a cascading failure. A Nest microservice client speaks in two registers, and choosing the wrong one is the first structural mistake. client.send(pattern, data) is request-response: it returns an Observable of a reply, and the caller waits for the handler (@MessagePattern) to compute and return an answer. client.emit(pattern, data) is event-based: it is fire-and-forget — the handler (@EventPattern) reacts, returns nothing, and the emitter does not wait.

// REQUEST-RESPONSE: you need an answer NOW, and you block on it
const total: number = await firstValueFrom(
  this.client.send('cart.total', { cartId }),   // ↔ @MessagePattern('cart.total')
);

// EVENT-BASED: you are announcing a fact; nobody is waited on
this.client.emit('order.placed', { orderId, total }); // ↔ @EventPattern('order.placed')

The senior rule is about coupling in time. Use send only when you genuinely need the answer to continue — a query whose result you’ll use on the next line. Use emit for “this happened” facts that other services react to on their own schedule. The trap is reaching for send everywhere because it feels like a function call: a chain of send → send → send across services quietly rebuilds a synchronous monolith on top of a message bus. Now you have all the operational cost of a distributed system — network latency stacked per hop, and cascading failure when any link is down — with none of the decoupling that justified splitting the services in the first place. emit is what lets the order service finish even when the email service is down.

Delivery semantics: the three guarantees, one of which is a lie

Every broker offers a delivery guarantee, and there are exactly three to know.

At-most-once is fire-and-forget with no redelivery: the message is sent, and if it’s lost in transit or the consumer dies before processing, it is simply gone. Fast, cheap, lossy. Acceptable for a metric tick; unacceptable for a payment.

At-least-once is the workhorse and the default for Kafka, NATS JetStream, and RabbitMQ: the broker keeps redelivering a message until the consumer acknowledges it. Nothing is ever lost — but the same message can be delivered more than once, because a redelivery fires whenever the broker isn’t sure the consumer finished.

Exactly-once is the one people ask for and the one you almost never actually get. True end-to-end exactly-once delivery across a network is, in the general case, impossible. What production systems actually build is at-least-once delivery + idempotent processing, which yields effectively-once: the message may arrive twice, but the second arrival has no additional effect. Say it precisely, because the imprecise version is what causes the double-charge: you do not get exactly-once delivery; you engineer exactly-once effect.

Together these three categories cover every broker you will encounter: at-most-once when loss is acceptable, at-least-once when loss is not, and effectively-once when the processing side absorbs duplicates with idempotency. Without that idempotency layer, at-least-once becomes “sometimes twice,” which is how the incident in the Hook happened.

Acknowledgements and the redelivery window

The duplicate is born in the gap between “work done” and “ack recorded.” The consumer does the work, then acknowledges: Kafka commits a consumer offset; NATS JetStream and RabbitMQ send an explicit ack. If the consumer crashes, or the ack is lost, before the broker records that acknowledgement, the broker redelivers — and the message is processed a second time.

This means the order of operations is the choice of which failure you accept, and there is no free option:

// ACK BEFORE processing → at-most-once in effect: a crash here LOSES the message
@EventPattern('payment.charge')
async handleBad(@Payload() data, @Ctx() ctx: KafkaContext) {
  await ctx.getConsumer().commitOffsets([/* ... */]); // committed first
  await this.charge(data); // if the pod dies here, the charge never happens, never retried
}

// ACK AFTER processing → at-least-once: a crash here DUPLICATES the message on redelivery
@EventPattern('payment.charge')
async handleGood(@Payload() data, @Ctx() ctx: KafkaContext) {
  await this.charge(data);                              // work first
  await ctx.getConsumer().commitOffsets([/* ... */]);  // crash before this → redelivered → charged twice
}

Acking before processing loses messages on a crash; acking after processing duplicates them on a crash. You cannot have neither. The mature systems pick ack-after (never lose) and pay for it with idempotency (never double-apply). Which is the whole point of the next section.

Idempotency: the fix, not a nicety

Because at-least-once will deliver duplicates, the consumer must be idempotent: processing the same message twice must produce the same end state as processing it once. This is not achieved by hoping; it is engineered with three moving parts:

1. A stable idempotency key that identifies the logical operation — carried on the message, the same across redeliveries (e.g. paymentId, or a client-supplied key). Not a random per-attempt id, or every redelivery looks new.
2. A dedup record of keys already processed — a processed_events table, checked first and written in the same transaction as the effect.
3. A side effect that is safe to repeat — guarded so the second arrival is a no-op.

@EventPattern('payment.charge')
async handleCharge(@Payload() data: { paymentId: string; amount: number }) {
  await this.dataSource.transaction(async (manager) => {
    // claim the key; if it already exists, this insert affects 0 rows → already processed
    const res = await manager.query(
      `INSERT INTO processed_events (key) VALUES ($1) ON CONFLICT (key) DO NOTHING`,
      [data.paymentId],
    );
    if (res.rowCount === 0) return; // duplicate redelivery → skip the charge entirely
    await this.charge(data); // first time only; commits atomically with the dedup row
  });
}

That ON CONFLICT (key) DO NOTHING is the load-bearing line: the dedup row and the charge commit in one transaction, so a redelivery either finds the key already there (and skips) or claims it and charges — never both. This is what the double-charge incident was missing. The fix shipped that day was exactly this: a unique idempotency key on the charge plus a processed-events check before calling Stripe.

Poison messages and the dead-letter queue

At-least-once has a second, sharper edge. Suppose a message can never succeed — malformed payload, a schema mismatch, a bug that throws on this one record. The broker’s redelivery, which saves you from transient crashes, now works against you: it redelivers the poison message forever. Worse, in an ordered stream the poison sits at the head and blocks every message behind it — head-of-line blocking — while burning CPU on an infinite retry loop.

The answer is a dead-letter queue (DLQ): after N failed attempts (commonly 3–5 retries), stop retrying inline and route the message to a separate queue for inspection and manual or automated handling, so the main stream keeps flowing. The retry count is the dial — too low and you DLQ transient blips that would have succeeded; too high and a poison message stalls the partition for minutes.

One more guarantee to keep honest: ordering is per-partition only. Kafka guarantees order within a partition (chosen by the message key), JetStream within a subject stream — but across partitions/keys there is no global order. If two events must be applied in sequence, they must share a partition key; otherwise “we’ll process them in order” is a guarantee you don’t actually have.