Dead Letter Queue

Tags

"When a messaging system cannot deliver a message, it moves the offending message to a dead message queue." — Hohpe & Woolf, Enterprise Integration Patterns, 2003

Intent

The Dead Letter Queue (DLQ) is an error-handling channel for undeliverable or unprocessable messages. When a consumer fails to process a message after exhausting all retry attempts, the message is routed to the DLQ rather than being discarded or causing the consumer to block indefinitely. The DLQ preserves the failed message for later inspection, reprocessing, or discarding — converting a silent failure into an auditable, recoverable event.

The full DLQ lifecycle has three phases: (1) Message arrives at consumer → consumer fails → retry policy exhausted → message routed to DLQ; (2) Operations team inspects DLQ → diagnoses root cause (bug, schema mismatch, downstream outage); (3) Message is retried (replayed to original topic), corrected and reprocessed, or discarded after a deliberate human decision.

Implementing a DLQ without also implementing recovery tooling and alerting is operationally equivalent to silent message loss — the DLQ becomes a black hole. Every DLQ deployment requires defined recovery paths and an alert on DLQ message count > 0.

When NOT to Use

  • For validation errors that should be rejected and never retried — use a validation error channel instead with immediate DLQ classification at the point of ingestion
  • As a substitute for fixing the root cause — DLQ is a safety net, not a permanent solution; unaddressed DLQ growth signals an unfixed bug
  • When messages are ephemeral and loss is acceptable (e.g., telemetry sampling) — DLQ overhead and operational burden are unnecessary in at-most-once delivery contexts

When to Use

  • Any message consumer that must not lose messages on processing failure
  • Event-driven systems where failed events must be auditable and recoverable
  • Saga failure handling — a failed saga step routes its event to the DLQ for compensation decision

How It Works

Consumer receives message → processing fails → message enters retry loop (with exponential backoff) → after N retries, message is published to the DLQ topic/queue → DLQ consumer or operator reads the DLQ → recovery decision made.

Key properties: the original message content must be preserved (use .useOriginalMessage() in Camel or equivalent) so the recovery operator sees the unmodified payload. Enrich the DLQ message with failure metadata: failure reason, exception message, retry count, original topic, and timestamp. This metadata drives triage — without it, every DLQ message requires a log correlation exercise.

Sequence Diagram

sequenceDiagram
    participant Q as Source Queue
    participant C as Consumer
    participant R as Retry Logic
    participant DLQ as Dead Letter Queue
    participant Op as Operator

    Q->>C: deliver message
    C->>C: process()
    Note right of C: Processing fails

    C->>R: retry with backoff
    R->>C: attempt 2
    C->>C: process()
    Note right of C: Fails again

    R->>C: attempt 3
    C->>C: process()
    Note right of C: Fails again (max retries)

    R->>DLQ: publish to DLQ
    Note right of DLQ: Original message preserved<br/>+ failure reason<br/>+ retry count<br/>+ timestamp

    Op->>DLQ: inspect failed messages
    Op->>Op: triage and fix
    Op->>Q: replay corrected message

    Note over Q,DLQ: Messages that exceed retry limit<br/>are routed to DLQ for manual recovery

TypeScript Example

// Dead Letter Queue — TypeScript (manual retry-then-DLQ pattern)
// Source: Hohpe & Woolf, EIP 2003; kafkajs-dlq pattern (github.com/Nevon/kafkajs-dlq)
const MAX_RETRIES = 3;
 
async function handleWithDLQ(
  message: { value: string },
  process: (msg: string) => Promise<void>,
  dlqProducer: { send: (args: { topic: string; messages: unknown[] }) => Promise<void> }
): Promise<void> {
  let attempt = 0;
  while (attempt < MAX_RETRIES) {
    try { await process(message.value); return; }
    catch { attempt++; }
  }
  // Retry exhausted — route to DLQ for manual inspection
  await dlqProducer.send({ topic: 'orders.dlq', messages: [{ value: message.value }] });
}

Java Example

// Dead Letter Channel — Apache Camel Java DSL
// Source: camel.apache.org/manual/error-handler.html
// Note: seda: is in-memory; production uses activemq:queue:orders.dlq or kafka:orders.dlq
public void configure() {
    errorHandler(deadLetterChannel("seda:orders.dlq")
        .maximumRedeliveries(3)
        .redeliveryDelay(1000)          // 1 second base delay between retries
        .useOriginalMessage());          // preserve original payload, not partially-processed
 
    from("seda:orders")
        .to("bean:orderProcessor");
}

Recovery Strategy

Every DLQ implementation requires three defined recovery paths:

  1. Retry — Replay the message to the original topic or queue. Fix the root cause first (bug fix, downstream service restoration). Requires Idempotent-Consumer on the target consumer — retry replay must not produce duplicate side effects.

  2. Manual Inspection — Operators examine the failed message, diagnose the failure reason (schema mismatch, downstream timeout, validation error), correct the payload or system state, then re-enqueue. Tooling required: DLQ consumer UI, message viewer, re-enqueue mechanism.

  3. Discard — After investigation confirms the message is unrecoverable or no longer relevant (references a deleted entity, expired event), acknowledge and remove. Requires explicit human decision — never auto-discard without logging the reason and operator identity.

Alerting is mandatory: Configure an alert on DLQ message count > 0. A DLQ with no consumer or alerting is operationally equivalent to message loss.

Lineage Backward

  • Domain-Events — Failed domain events are the primary DLQ candidates in event-driven systems. When a domain event handler fails after exhausting retries, the event is preserved in the DLQ for later compensation or replay.

Lineage Forward

  • Choreography-Saga-Pattern (Phase 13) — Failed saga steps route their triggering event to the DLQ. The saga coordinator reads the DLQ to decide whether to retry, trigger compensation, or escalate.
  • Idempotent-Consumer — Retry recovery (path 1 above) replays the message to the original consumer. That consumer MUST be idempotent to avoid double-processing side effects.
ConceptRelationship
Idempotent-ConsumerRetry recovery from the DLQ replays to the original consumer — that consumer must be idempotent
Domain-EventsDomain event handlers are the primary DLQ candidates in event-driven systems
Choreography-Saga-PatternSaga failure handling routes failed step events to the DLQ for compensation decision
Message-RouterRoutes failed messages to the DLQ channel based on processing outcome
  • Message-Queue — Message-Queue covers infrastructure delivery guarantees; Dead-Letter-Queue covers the error-handling channel built on top; they operate at different layers of the same async messaging stack
  • Notification-System-Design — notifications exhausting their retry budget are moved to the DLQ for manual review; the three DLQ recovery paths (retry, discard, manual inspection) apply directly
  • Bloom-Filter — a Bloom filter can serve as a fast pre-check before the expensive DLQ deduplication store lookup, reducing database reads for duplicate message detection
  • Web-Crawler-Design — crawl jobs that fail after N retries can be routed to a DLQ for operator review rather than silently dropped; the same DLQ recovery patterns apply

Sources

Linked mentions