Pipes and Filters

Tags

"Use a Pipe and Filter architectural style to divide a larger processing task into a sequence of smaller, independent processing steps (Filters) that are connected by channels (Pipes)." — Hohpe & Woolf, Enterprise Integration Patterns, 2003

Intent

Pipes and Filters decomposes a large processing task into a sequence of independent, reorderable filter stages connected by pipes (channels). Each filter has one inbound pipe and one outbound pipe. Filters are independent — they can be rearranged, replaced, or added without modifying other filters. The message flows through every filter stage sequentially; no stage is skipped.

This pattern differs from Decorator-Pattern in a fundamental way: Decorator adds behaviour to a single object by wrapping it; Pipes and Filters routes a message through independent processing stages. A Decorator wraps an object and is permanently attached to it; a Filter transforms a message in transit and is interchangeable with other filters. Decorators accumulate on one object; Filters are stateless and composable. The pattern also differs from Chain-of-Responsibility-Pattern: CoR routes the message to the first handler that matches and stops; Pipes and Filters passes the message through ALL filter stages sequentially — every filter processes every message.

Canonical use cases include ETL pipelines (extract → validate → transform → load), request processing middleware (decrypt → authenticate → enrich → log → persist), and stream processing frameworks. Apache Camel and Node.js streams implement this pattern natively as a first-class routing primitive.

When NOT to Use

  • Single-step transformation — a direct processor is simpler and has no pipeline overhead
  • Filters share mutable state — breaks independence; use a stateful aggregate processor instead
  • Error handling requires rollback of previous stages — use Choreography-Saga-Pattern or a distributed transaction instead
  • Pipeline stages must execute transactionally — filters are independently stateless by design; transactional semantics require a different approach (e.g., unit-of-work boundary wrapping all stages)

When to Use

  • Multi-step message enrichment chains (decrypt → validate → enrich → persist)
  • ETL pipelines where stages can be reordered or replaced independently
  • Request processing middleware (similar to Express.js or ASP.NET middleware chains)
  • When each filter step should be independently testable in isolation
  • When filters may need to be reordered without modifying surrounding code

How It Works

The pipeline structure is: Pump (message source) → [Pipe → Filter → Pipe] × N → Sink (final destination).

  • Pump: Produces or retrieves the initial message (message queue consumer, HTTP request, file reader)
  • Pipe: The channel connecting two filters — can be a function call, queue, stream, or async iterator
  • Filter: Receives a message from its inbound pipe, applies a transformation, and sends the result to its outbound pipe
  • Sink: The final consumer of the processed message (database writer, response sender, downstream producer)

Filters are stateless — they do not know about other filters in the pipeline. This independence enables parallel development, independent testing, and runtime reconfiguration of the pipeline order.

Flow Diagram

flowchart LR
    P["Pump<br/>(message source)"]
    F1["Filter A<br/>validate"]
    F2["Filter B<br/>transform"]
    F3["Filter C<br/>enrich"]
    S["Sink<br/>(final destination)"]

    P -->|"pipe"| F1
    F1 -->|"pipe"| F2
    F2 -->|"pipe"| F3
    F3 -->|"pipe"| S

    style P fill:#e1f5fe,stroke:#0288d1
    style S fill:#e8f5e9,stroke:#388e3c
    style F1 fill:#fff3e0,stroke:#f57c00
    style F2 fill:#fff3e0,stroke:#f57c00
    style F3 fill:#fff3e0,stroke:#f57c00

TypeScript Example

// Pipes and Filters — TypeScript (function pipeline)
// Source: Hohpe & Woolf, Enterprise Integration Patterns, 2003
type Filter<T> = (message: T) => T;
 
function pipeline<T>(message: T, ...filters: Filter<T>[]): T {
  return filters.reduce((msg, filter) => filter(msg), message);
}
 
// Usage: independent, composable filters
const decrypt  = (msg: string) => msg.replace('[enc]', '');
const validate = (msg: string) => { if (!msg) throw new Error('empty'); return msg; };
const enrich   = (msg: string) => msg + '[enriched]';
 
const result = pipeline(message, decrypt, validate, enrich);

Java Example

// Pipes and Filters — Apache Camel Java DSL
// Source: camel.apache.org/components/4.x/eips/enterprise-integration-patterns.html
from("direct:orders")
    .to("direct:decrypt")
    .to("direct:validate")
    .to("direct:enrich")
    .to("direct:persist");
// Each direct: endpoint is an independent filter (Processor or RouteBuilder)

Lineage Backward

  • Decorator-Pattern — Decorator adds behaviour to a single object by wrapping it; Pipes and Filters routes a message through independent processing stages. Key distinction: Decorator modifies an object's behaviour; Filters transform a message in transit. Filters can be reordered or bypassed; Decorator wrappers cannot be rearranged without restructuring the wrapping chain.
  • Chain-of-Responsibility-Pattern — CoR routes the request to the first handler that matches and stops processing; Pipes and Filters passes the message through ALL filter stages sequentially. CoR is about finding a single responsible handler; Pipes and Filters is about composing a processing sequence.

Lineage Forward

  • Sidecar-Pattern (Phase 14) — Sidecar implements Pipes and Filters at the infrastructure layer: network traffic passes through proxy filter stages (authentication, logging, rate limiting, circuit breaking) before reaching the application container. The sidecar proxy is a Filter in the network pipe.
PatternRelationship
Decorator-PatternDecorator wraps a single object to add behaviour; Pipes and Filters routes a message through independent, reorderable stages
Chain-of-Responsibility-PatternCoR stops at first matching handler; Pipes and Filters passes through ALL filter stages
Message-RouterMessage Router routes a message to a destination channel without transforming it; Pipes and Filters transforms the message as it passes through each stage
Sidecar-PatternSidecar is Pipes and Filters at the infrastructure layer — network traffic as the "pipe", proxy interceptors as "filters"
  • Hexagonal-Architecture — A Pipes-and-Filters pipeline can be implemented as a series of driven adapters in Hexagonal Architecture, each stage receiving input through a secondary port and passing output to the next.
  • gRPC-Service-Design — gRPC interceptor chains execute as processing pipelines; each interceptor is a filter stage — the Pipes-and-Filters pattern applied at the RPC protocol layer
  • Search-Autocomplete-Design — the top-K aggregation pipeline (log collection, aggregation, reduction, store update) is a classic Pipes-and-Filters architecture; each stage is independently scalable and testable
  • Message-Queue — a pipeline of Pipes-and-Filters stages can be decoupled by placing a message queue between stages; the queue provides buffering and backpressure so each filter processes at its own pace

Sources

Linked mentions