Message Queue

Infrastructure component that decouples producers from consumers by buffering messages for asynchronous delivery, enabling systems to handle traffic bursts, decouple services, and support fan-out to multiple consumers.

Scope note: This note covers infrastructure-layer message queuing — broker topology, delivery guarantees, and backpressure. Application-layer reliability patterns that operate on top of a queue are out of scope here: see Dead-Letter-Queue for error channel and recovery strategy, Idempotent-Consumer for duplicate message handling, and Choreography-Saga-Pattern for event-driven distributed workflow coordination.

When NOT to Use

• Request-response patterns requiring synchronous results — queue introduces latency and requires callback or polling; use a direct HTTP/gRPC call instead
• Tiny workloads where the broker infrastructure cost exceeds benefit — use an in-process task queue (BullMQ in-process, Java ExecutorService) rather than deploying a full broker
• When exactly-once semantics are required across heterogeneous systems — Kafka EOS only guarantees exactly-once within the Kafka cluster; cross-system exactly-once requires distributed transactions

Core Mechanism

Point-to-Point vs Pub/Sub

Two distinct messaging topologies serve different fan-out requirements.

Point-to-point (queue): One message is delivered to exactly one consumer. The consumer removes the message from the queue after processing. Multiple competing consumers can subscribe to the same queue for horizontal scale — but each message is processed by only one of them. Use when exactly one service should act on each event (e.g., an order processing worker).

Pub/sub (topic/channel): One message is delivered to N consumer groups, each receiving its own independent copy. Each consumer group processes independently at its own pace. Use when multiple services must react to the same event (fan-out), such as an OrderPlaced event consumed by inventory, billing, and shipping services simultaneously.

Implementation note: Kafka topics are pub/sub by default — each consumer group maintains its own offset, so the same message is independently processed by all groups. RabbitMQ queues are point-to-point by default; a fanout exchange with multiple queue bindings achieves pub/sub.

Backpressure

When consumer throughput < producer throughput, queue depth grows. Without backpressure, queue memory grows unbounded leading to broker out-of-memory (OOM) conditions.

Backpressure mechanisms:

• Producer-side rate limiting: Block or return an error to the producer when the queue depth exceeds a configured threshold. Forces the producer to slow down rather than let the queue overflow.
• Consumer-side pull (Kafka native): Consumers pull batches from the broker at their own pace. The broker never pushes more than the consumer requests, providing natural backpressure without explicit configuration.
• Dead-letter diversion: Age-out old messages to a DLQ to prevent indefinite queue depth growth when consumer lag is a recurring problem. This is a pressure relief valve, not a substitute for fixing consumer throughput.

Component Diagram

Message-Queue-diagram.excalidraw

Key Variants

Three delivery guarantees represent different tradeoffs between reliability and overhead.

At-most-once: Producer fires and forgets; the broker does not persist or retry the message; consumer processes without acknowledgement. Risk: message loss — if the broker or consumer crashes before processing, the message is gone. Benefit: lowest latency, zero overhead. Use for metrics/telemetry sampling, best-effort notifications, or any workload where occasional message loss is acceptable.

At-least-once: Broker persists the message. Consumer must explicitly acknowledge (commit its offset) after successful processing; the broker re-delivers on consumer failure or timeout. Risk: duplicate delivery — if the consumer crashes after processing but before committing, the message is redelivered. Application must handle idempotency (see Idempotent-Consumer). Benefit: no message loss. Default for Kafka, SQS. Most common production choice for event-driven systems.

Exactly-once: Broker and consumer coordinate to ensure exactly one delivery with exactly one processing effect. Kafka achieves this via producer idempotency (each producer has a unique ID; broker deduplicates by sequence number) combined with transactional consumer commit (atomic offset commit + downstream write within a Kafka transaction). High overhead; correctness guarantee applies only within the Kafka cluster boundary — cross-system exactly-once requires distributed transactions. Use only when duplicate effects are catastrophic and cannot be handled at the application layer.

Design Decisions

Use a message queue when:

• Producer and consumer are decoupled with different availability windows — the queue absorbs bursts when the consumer is slow or offline
• Processing is slow relative to ingestion — queue provides the buffer for burst absorption
• Fan-out to multiple consumers is required — pub/sub topology distributes one event to N services
• Work queue pattern needed — many independent tasks distributed across a pool of workers

Use a synchronous call instead when:

• The caller needs an immediate response — queue-based callback/polling adds complexity and latency
• Transaction semantics are required across both operations — a queue hop breaks transactional boundaries
• The latency budget is tight and the queue hop overhead is unacceptable — e.g., sub-10ms user-facing operations

Infrastructure product comparison (architectural level only):

Pitfalls

Skeletal Snippet

// Message queue producer/consumer — teaching example (at-least-once delivery)
// Source: Alex Xu, System Design Interview Vol. 1, Ch. 10 (notification system)
// PRODUCTION NOTE: Use KafkaJS, @aws-sdk/client-sqs, or amqplib with proper error handling
 
// Producer: fire-and-forget (at-least-once if broker persists)
async function publish(topic: string, message: Record<string, unknown>): Promise<void> {
  await broker.send({ topic, messages: [{ value: JSON.stringify(message) }] });
  // No ack wait = at-most-once. Await ack from broker = at-least-once.
}
 
// Consumer: commit offset only after successful processing (at-least-once)
async function consume(topic: string): Promise<void> {
  await broker.subscribe({ topic });
  await broker.run({
    eachMessage: async ({ message, commitOffsets }) => {
      await processMessage(JSON.parse(message.value!.toString()));
      await commitOffsets();   // commit after processing = at-least-once
      // Committing BEFORE processing = at-most-once (offset advances even if processing fails)
    },
  });
}

Existing Pattern Connections

• Dead-Letter-Queue — DLQ is an application-layer pattern built on top of queue infrastructure; Message-Queue covers the infrastructure broker topology and delivery guarantees; Dead-Letter-Queue covers the error-handling channel and recovery paths
• Idempotent-Consumer — at-least-once delivery (the infrastructure default for Kafka and SQS) requires consumers to be idempotent at the application layer; Message-Queue documents the delivery guarantee; Idempotent-Consumer documents the consumer design that handles duplicate messages safely
• Choreography-Saga-Pattern — choreography sagas use pub/sub topics as the event bus; Message-Queue covers the infrastructure topology (fan-out to N consumer groups); Choreography-Saga covers the distributed workflow coordination pattern built on top of that topology
• Pipes-and-Filters — a pipeline of Pipes-and-Filters stages can be decoupled by placing a message queue between stages; the queue provides buffering and backpressure so each filter processes at its own pace

Backlinks