Delivery semantics, deduplication and the outbox pattern

Why it matters

Asynchronous messaging is how systems absorb load spikes and decouple teams, and it is also where duplicate charges, double emails and lost orders come from. "We use a queue, so it is reliable" invites the follow-up: what happens when the consumer crashes after doing the work but before acknowledging it? The answer always combines delivery semantics with idempotency, so you need both.

Core ideas

Delivery semantics follow from acknowledgment order.

• At-most-once: acknowledge (or commit the offset) before processing. A crash mid-processing loses the message, but nothing is processed twice.
• At-least-once: process, then acknowledge. A crash after processing but before the acknowledgment causes redelivery, so duplicates are normal.
• Exactly-once delivery over an unreliable network is not achievable as a transport property. What systems offer is effectively-once processing: at-least-once delivery plus deduplication, or an atomic commit of results and progress, within a defined boundary.

Where duplicates and reordering come from. Producers retry after timeouts with unknown outcomes; brokers redeliver after visibility timeouts or consumer-group rebalances; consumers crash between side effect and acknowledgment. Order holds at most within one partition or message group, and retries or parallel consumers can still reorder unless processing is serialized per key.

Idempotent consumers. Make processing safe to repeat:

• Prefer naturally idempotent writes (set status to SHIPPED, upsert by primary key) over increments.
• Give every message or request a unique ID (an idempotency key chosen by the producer), record processed IDs in the same transaction as the effect, and skip IDs already recorded. Retain IDs at least as long as redelivery is possible.
• For external side effects (an email, a charge), pass the key to the provider if it supports one, or record intent and outcome so a retry can check before acting.

The dual-write problem. A service that updates its database and then publishes an event can crash between the two, or have one succeed and the other fail, leaving them inconsistent. Distributed transactions spanning a database and a broker are rarely available or worth it. The transactional outbox fixes this: in the same local transaction as the state change, insert the event into an outbox table; a relay (a poller, or change data capture on the database log) publishes outbox rows and marks them sent. The relay may publish a row twice, so consumers still deduplicate.

BEGIN
  UPDATE orders SET status = 'PAID' WHERE id = 42
  INSERT INTO outbox(event_id, topic, payload) VALUES ('e-9f1', 'order-paid', ...)
COMMIT
relay:    read unsent outbox rows -> publish -> mark sent        (at-least-once)
consumer: if event_id already processed: skip
          else: apply effect and record event_id in one transaction

Consumer offsets. In a partitioned log, each consumer group tracks a position per partition. Committing before processing gives at-most-once; committing after gives at-least-once. Storing the offset in the same database transaction as the results makes the pair atomic, which is effectively-once for that sink.

What "exactly-once" means in a log broker. Idempotent producers attach sequence numbers so the broker drops retried duplicates within a producer session. Transactions let a read-process-write job atomically write its output records together with its input offsets, and consumers configured to read only committed data never see aborted output. The boundary ends at the broker: a database row written or an API called by the consumer is outside it and needs its own idempotency. Managed queues make narrower promises still; one deduplicates only within a fixed time window.

Common misconceptions

• "Our broker guarantees exactly-once, so consumers need not be idempotent." The guarantee covers a specific pipeline inside the broker; anything touching an external system can repeat.
• "Acknowledging after processing avoids duplicates." It avoids loss and causes duplicates on crash; idempotency removes their effect.
• "Publishing right after the commit is good enough." The process can die between the two; use an outbox or change data capture.
• "A queue preserves order." Usually only per partition or message group, and only with one in-flight message per key.
• "Deduplication keys can be kept forever." Retention must cover the redelivery horizon; choose it explicitly and expire the rest.

How to approach it

Read DDIA Ch. 11 (messaging systems, partitioned logs and fault tolerance), then the Kafka design notes on delivery semantics and the outbox pattern description. Write consumer pseudocode for both offset orders and mark the crash points that cause loss or duplicates. Graduates should know the three semantics and why idempotency matters. Mid-level candidates should design an outbox and idempotent consumers. Senior candidates should define the exactly-once boundary, per-key ordering and the operational handling of poison messages and backlogs.