Distributed Cache

In-memory data store distributed across multiple nodes, providing sub-millisecond read latency for frequently accessed data by keeping hot data closer to the application than the primary database.

Scope note: This note covers infrastructure-layer caching — where and how data is cached in a distributed system. The CQRS read model is a use case of caching (a materialised projection stored in a cache or separate data store), but CQRS itself is an architectural pattern that operates independently of whether caching is involved. See CQRS-Pattern for the read/write model separation pattern. This note does not re-explain CQRS projections.

When NOT to Use

• Data requires strong consistency (financial ledgers, inventory counts where stale data causes incorrect business decisions) — use DB with read replicas instead; a cache-aside or write-behind strategy introduces an eventual consistency window that is unacceptable for these workloads
• Data is write-heavy and rarely read — cache hit rate will be low; the overhead of maintaining the cache (memory, network round-trips, invalidation) exceeds the benefit; the database handles write throughput directly
• Dataset fits in application memory already — an in-process cache (application-level Map or Guava Cache) is simpler and faster; a distributed cache adds a network hop for every access, which costs 0.5–5ms vs nanoseconds for an in-process lookup

Core Mechanism

The standard cache-aside read path:

Client → Application → Cache (hit? return data)
                     ↓ (miss)
                     → Database → return data → populate cache → return to client

Application checks the cache before the database on every read. A cache miss triggers a database read, which populates the cache for subsequent requests. Cache entries expire via TTL (time-based) or are evicted via eviction policy (space-based). Data is distributed across cache nodes via consistent hashing — each key maps to a position on a ring that resolves to a specific cache node, minimising remapping when nodes are added or removed.

Component Diagram

Distributed-Cache-diagram.excalidraw

Key Variants

Caching strategies — the write path determines cache-database consistency:

Design Decisions

Eviction Policies

When the cache reaches memory capacity, an eviction policy determines which entries are removed:

• LRU (Least Recently Used): evict the entry not accessed for the longest time — default for most workloads; good approximation of "least likely to be needed again"
• LFU (Least Frequently Used): evict the entry accessed least often — better for stable working sets with a known hot subset; higher bookkeeping overhead than LRU
• TTL (Time-To-Live): evict after a fixed duration regardless of access frequency — use for time-sensitive data (sessions, authentication tokens, OTP codes) where data has a natural expiry
• Random: evict a randomly selected entry — simple; no bookkeeping; acceptable for uniform-access workloads where all entries have equal likelihood of future access

Redis vs Memcached

Failure Modes

Skeletal Snippet

// Cache-aside (lazy loading) — teaching example
// Source: Alex Xu, System Design Interview Vol. 1, Ch. 6
// PRODUCTION NOTE: Use Redis client (ioredis/redis-om) with proper TTL and error handling
async function getUser(userId: string, cache: Cache, db: DB): Promise<User> {
  const cached = await cache.get(`user:${userId}`);
  if (cached) return JSON.parse(cached);      // cache hit
 
  const user = await db.query(
    'SELECT * FROM users WHERE id = $1', [userId]
  );
  await cache.set(
    `user:${userId}`,
    JSON.stringify(user),
    'EX', 3600            // 1-hour TTL — tune per staleness tolerance
  );
  return user;
}
// Write path (cache-aside): write DB only; cache populated on next read.
// Write-through: call cache.set() in the write path after DB write.

Cache Warming and Cold Start

After a cache restart or deployment, all entries are absent (cold cache). Every request misses until the cache is warm. Strategies:

• Lazy warming: accept degraded performance at startup; the cache warms naturally as traffic hits the DB and populates entries. Simple; causes a traffic spike on the DB at cold start.
• Pre-warming: proactively populate the cache with top-N keys (e.g., most popular products, active user sessions) before accepting traffic. Requires a warm-up job; reduces cold start DB load.
• TTL staggering: even after warm-up, ensure TTL values are staggered (see Cache avalanche above) to prevent simultaneous expiry once the warm cache starts to age out.

Cache Topology: Cluster vs Replication

Distributed caches can be deployed in two topologies:

• Sharded cluster (Redis Cluster, Memcached consistent hash): each node owns a subset of keys; capacity scales horizontally by adding nodes; a node failure loses its key subset until re-populated
• Replicated (Redis Sentinel, Redis primary-replica): all nodes hold all data; any node can serve reads; write goes to primary and replicates to replicas; capacity is limited to single-node memory but read throughput scales with replicas; higher durability

Existing Pattern Connections

• CQRS-Pattern — the CQRS read model is often materialised into a distributed cache (a timeline cache, a user profile cache, a search results cache); this note covers the infrastructure layer (how caching works); CQRS-Pattern covers the architectural separation of read and write models
• Bloom-Filter — a Bloom filter can serve as a probabilistic pre-check before cache lookup to prevent cache penetration: if the filter says a key definitely does not exist in the database, the cache and database are both bypassed; Phase 28 note, now in vault
• CAP-Theorem — write-through, cache-aside, and write-behind represent different positions on the consistency-availability spectrum; write-through is a CP choice (consistency at the cost of write latency); cache-aside with TTL is an AP choice (availability with eventual consistency); CAP-Theorem provides the formal framework
• lru-cache — the O(1) doubly linked list + hash map data structure that implements the LRU eviction policy described in this note; see the algorithm note for implementation details, sentinel node patterns, and edge cases

Backlinks