Idempotent Consumer

Tags

"Design a receiver to be an Idempotent Receiver — one that can safely receive the same message multiple times." — Hohpe & Woolf, Enterprise Integration Patterns, 2003

Intent

The Idempotent Consumer pattern ensures that processing a message multiple times produces the same result as processing it once. This is a requirement in any system that retries failed messages — DLQ replay, at-least-once delivery guarantees, and saga compensation retries all create scenarios where a consumer may receive the same message more than once. Without idempotency, these duplicates cause double-charging, duplicate records, incorrect aggregate state, or other data corruption.

Two strategies achieve idempotency: (1) Explicit deduplication — track processed message IDs in a persistent store; check before processing, record after success; reject duplicates silently. (2) Semantic idempotency — design the operation to be naturally repeatable (e.g., UPDATE SET status='shipped' WHERE status='pending' is idempotent; INSERT INTO shipments is not). Prefer semantic idempotency when possible — it requires no deduplication store and handles failures without coordination overhead.

Distinguish idempotent consumer from Kafka's Exactly-Once Semantics (EOS): EOS is a broker-level guarantee about delivery count; idempotent consumer is an application-layer guarantee about processing effects. They are complementary, not interchangeable. EOS has performance overhead; application-level idempotency is pragmatic and works across any messaging system. Use EOS where the broker supports it; use idempotent consumer design regardless.

When NOT to Use

  • When at-most-once delivery is acceptable and message loss is tolerable — idempotency adds overhead (persistent store, lookup on every message) that is unnecessary if duplicates are impossible by design
  • When the operation is already naturally idempotent (e.g., cache warm-up, read-only queries, setting a flag to a fixed value) — explicit deduplication is redundant
  • In-memory deduplication only — an in-memory Set is lost on process restart, providing false safety (see Production Note in TypeScript example below)

When to Use

  • Any consumer in an at-least-once delivery system (Kafka default, RabbitMQ with requeue, SQS)
  • DLQ retry replay — when a message from the Dead-Letter-Queue is replayed to the original consumer
  • Saga compensating transactions — compensation steps must not execute twice on retry
  • Event sourcing event handlers — replaying events for projection rebuild must be idempotent

How It Works

Strategy 1 — Explicit deduplication: receive message → check persistent store for messageId → if found, skip (acknowledge without processing) → process → persist messageId. The persistent store must survive restarts: a DB table with a unique constraint on (messageId, consumerId) or a Redis key with TTL. Record the ID after successful processing, not before — recording before risks marking a message as processed when processing fails.

Strategy 2 — Semantic idempotency: design the operation so repeated execution has no additional effect. Examples: UPSERT, UPDATE ... WHERE status = 'pending', set membership checks. Prefer this when the domain operation can be designed this way — no deduplication store needed.

Idempotency vs Exactly-Once: Kafka EOS (transactions + idempotent producer) guarantees broker-level exactly-once delivery; application-layer idempotency guards against processing failures and retries from DLQ or saga compensation. They are defence-in-depth layers — EOS prevents broker-level duplicates; idempotent consumer prevents application-level double-processing.

Sequence Diagram

sequenceDiagram
    participant Q as Message Queue
    participant C as Consumer
    participant S as Dedup Store
    participant P as Processor

    Q->>C: deliver message (id=msg-42)
    C->>S: lookup(msg-42)
    S-->>C: NOT FOUND

    C->>P: process(message)
    P-->>C: success
    C->>S: store(msg-42)
    C->>Q: acknowledge

    Note over Q,S: First delivery: processed normally

    Q->>C: deliver message (id=msg-42)
    Note right of Q: Redelivery (broker retry<br/>or network issue)
    C->>S: lookup(msg-42)
    S-->>C: FOUND

    C->>Q: acknowledge (skip processing)

    Note over Q,S: Duplicate delivery: skipped<br/>via deduplication store lookup

TypeScript Example

// Idempotent Consumer — TypeScript (Set-based deduplication)
// Source: Hohpe & Woolf, Enterprise Integration Patterns, 2003
// PRODUCTION NOTE: Replace Set with persistent store — DB unique constraint on
// (messageId, consumerId) or Redis key with TTL. In-memory Set is lost on restart.
const processedIds = new Set<string>();
 
async function idempotentConsume(
  message: { messageId: string; payload: unknown },
  process: (payload: unknown) => Promise<void>
): Promise<void> {
  if (processedIds.has(message.messageId)) return; // duplicate — skip
  await process(message.payload);
  processedIds.add(message.messageId);             // record after success (not before)
}

Java Example

// Idempotent Consumer — Apache Camel Java DSL
// Source: cwiki.apache.org/confluence/display/CAMEL/Idempotent+Consumer
// PRODUCTION NOTE: Replace MemoryIdempotentRepository with
// JdbcMessageIdRepository or RedisIdempotentRepository for crash-safe deduplication
from("direct:inbox")
    .idempotentConsumer(
        header("messageId"),
        MemoryIdempotentRepository.memoryIdempotentRepository(500)
    )
    .to("bean:orderProcessor");
// Duplicate messages (same messageId) are silently skipped

Lineage Backward

  • Domain-Events — Domain event handlers are common idempotent consumer implementations. In event sourcing, projection handlers must be idempotent to support event replay.

Lineage Forward

  • Compensating-Transactions (Phase 13) — Saga compensating transactions depend on idempotent consumers. If a saga step fails and the compensating transaction is retried, the compensation action must not execute twice. An idempotent compensating transaction is a prerequisite for correct saga failure recovery.
  • Dead-Letter-Queue — Retry recovery from the DLQ replays the original message to its consumer. That consumer must be idempotent to avoid double-processing side effects.
ConceptRelationship
Dead-Letter-QueueRetry replay from the DLQ requires an idempotent target consumer
Compensating-TransactionsSaga prerequisite — compensation steps must not execute twice on retry
Domain-EventsDomain event handlers are the canonical idempotent consumer implementation
Kafka EOSBroker-layer complement — prevents delivery duplicates; idempotent consumer prevents processing duplicates
AggregateApplication-layer semantic idempotency uses aggregate state checks before applying changes
  • Operational-API-Patterns — HTTP idempotency keys (client-driven UUID header) are the HTTP-layer counterpart to messaging-layer Idempotent Consumer; Operational-API-Patterns documents the distinction explicitly
  • Message-Queue — at-least-once delivery (the infrastructure default) requires idempotent consumers; Message-Queue documents the delivery guarantee; Idempotent-Consumer documents the consumer design that handles duplicates safely
  • Notification-System-Design — notification workers are idempotent consumers; provider idempotency key equals notification_id; the deduplication store pattern prevents duplicate cross-channel delivery
  • Web-Crawler-Design — content fingerprint deduplication is an application of the Idempotent-Consumer pattern: re-fetching the same content is a no-op by design
  • Chat-System-Design — re-delivered messages on reconnect are deduplicated by seq_id; the client is an idempotent consumer of the message stream, avoiding duplicate display

Sources

Linked mentions