Metrics and Dashboards

Metrics are aggregated numerical measurements of system behaviour over time. They differ from the other two observability pillars in a fundamental way: a metric discards the individual event and retains only the aggregate — a counter, a gauge value, a histogram bucket count. This makes metrics cheap to store, fast to query, and easy to alert on, but they cannot answer why a specific request failed.

• Metrics — aggregated numbers over time windows; ideal for alerting, SLO tracking, and capacity planning
• Logs — discrete event records with full context; ideal for post-incident investigation
• Traces — end-to-end request journeys with span timing; ideal for latency attribution across services

Dashboards make metrics actionable by placing the right aggregations in front of the right operator at the right moment.

When NOT to Use

Metrics alone are insufficient for:

• Debugging a specific request failure — metrics can tell you the error rate spiked; they cannot show which request failed or what the stack trace was. Use distributed traces (Distributed-Tracing) and structured logs for that.
• Understanding why an error occurred — you need logs and traces that carry per-request context.
• Verifying business-logic correctness — a zero-error-rate metric does not mean the order total was calculated correctly.

Cardinality Explosion Warning

Measurement Frameworks

Three frameworks cover the vast majority of instrumentation needs. Each targets a different system type.

RED Method (Request-oriented Services)

Origin: Tom Wilkie / Weaveworks (2018).

Target systems: HTTP services, gRPC services, message consumers, async workers.

Key question: Is my service handling requests correctly and fast enough?

USE Method (Resource-oriented Infrastructure)

Origin: Brendan Gregg (2012).

Target systems: CPUs, memory, disks, network interfaces, connection pools, thread pools.

Key question: Is a resource becoming a bottleneck?

Four Golden Signals (Google SRE)

Origin: Google SRE Book, Chapter 6.

Target systems: user-facing services where SLO adherence matters.

Key question: Is my service meeting user-visible quality targets?

Four Golden Signals is similar to RED but elevates Saturation to a first-class signal alongside Latency/Traffic/Errors, and explicitly distinguishes error latency from success latency.

Framework Comparison Table

Selection Flowchart

flowchart TD
    A[What are you measuring?] --> B{User-facing HTTP or gRPC service?}
    B -- Yes --> C[Use RED\nRate · Errors · Duration]
    B -- No --> D{Infrastructure resource?\nCPU / memory / disk / pool?}
    D -- Yes --> E[Use USE\nUtilization · Saturation · Errors]
    D -- No --> F{SRE / SLO-driven service?}
    F -- Yes --> G[Use Four Golden Signals\nLatency · Traffic · Errors · Saturation]
    F -- No --> H{Complex service needing both\nservice-level and resource-level view?}
    H -- Yes --> I[RED for service layer\n+ USE for resource layer]
    H -- No --> J[Start with RED —\nextend as needed]

Metric Types Reference

These are conceptual types, not API reference. For the Java API (Counter.builder, Timer.builder, Gauge), see Micrometer.

Decision rule: Prefer Histogram over Summary for latency. Prometheus computes percentiles server-side from histogram buckets, and histograms aggregate correctly across replicas. Summaries aggregate incorrectly — the p99 of a fleet is not the average of per-instance p99s.

TypeScript Example — prom-client 15.x

The following snippet instruments an Express service using the RED method. All labels are low-cardinality.

import { Registry, Counter, Histogram, Gauge } from 'prom-client'; // prom-client 15.x
import { Request, Response, NextFunction } from 'express';
 
const registry = new Registry();
 
// RED: Rate + Errors (Counter) — low-cardinality labels only
const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
  registers: [registry],
});
 
// RED: Duration (Histogram) — p50/p95/p99 computed by Prometheus from buckets
const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request latency',
  labelNames: ['method', 'route'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [registry],
});
 
// USE: Saturation (Gauge) — connection pool pressure
const activeConnections = new Gauge({
  name: 'db_pool_active_connections',
  help: 'Active database pool connections',
  registers: [registry],
});
 
export function metricsMiddleware(req: Request, res: Response, next: NextFunction): void {
  const end = httpDuration.startTimer({
    method: req.method,
    route: req.route?.path ?? 'unknown', // route template, NOT req.url — avoids high cardinality
  });
  res.on('finish', () => {
    httpRequests.inc({
      method: req.method,
      route: req.route?.path ?? 'unknown',
      status_code: String(res.statusCode),
    });
    end();
  });
  next();
}
 
// Expose /metrics endpoint for Prometheus scrape
export async function metricsHandler(_req: Request, res: Response): Promise<void> {
  res.set('Content-Type', registry.contentType);
  res.end(await registry.metrics());
}

Key decisions above:

• req.route?.path (template like /api/orders/:id) instead of req.url (actual URL like /api/orders/abc-123) — prevents per-request cardinality explosion
• buckets array follows the Prometheus default recommendations, adjusted downward for typical API latency

Java Example — Micrometer / Spring Boot

The following snippet instruments a Spring Boot service using the Micrometer Observation API, which produces both metrics and traces from a single @Observed annotation. For Counter.builder, Timer.builder, and Gauge API details, see Micrometer.

@Service
public class OrderService {
 
    private final ObservationRegistry observationRegistry;
    private final Counter ordersFailed;
 
    public OrderService(ObservationRegistry registry, MeterRegistry meterRegistry) {
        this.observationRegistry = registry;
        // low-cardinality tag "reason" — bounded set of validation failure categories
        this.ordersFailed = Counter.builder("orders.failed")
            .description("Orders that failed processing")
            .tag("reason", "validation")
            .register(meterRegistry);
    }
 
    // @Observed auto-records: rate (count), duration (timer), and errors
    // produces both a metric AND a trace span — single instrumentation point
    @Observed(name = "order.process", contextualName = "processOrder")
    public Order processOrder(OrderRequest request) {
        return repository.save(mapToOrder(request));
    }
 
    public void rejectOrder(OrderRequest request, String reason) {
        ordersFailed.increment(); // explicit RED Errors counter
        throw new ValidationException("Rejected: " + reason);
    }
}

For Counter.builder, Timer.builder, and Gauge API details, see Micrometer.

Dashboard Design Principles

Golden rule: Every panel must answer a specific operational question. If you cannot state the question in one sentence, the panel does not belong on the dashboard.

Layout Structure

Recommended three-row layout:

Rationale for separation: Row 1 and Row 2 serve different audiences (on-call engineer vs capacity planner). Mixing them forces both audiences to filter noise.

Principles

• Percentiles over averages — p99 latency reveals tail problems; the arithmetic mean hides them. A single 5-second request in a batch of 1ms requests yields a 50ms average that looks healthy.
• Error rate before error count — "0.1% error rate" is more actionable than "500 errors/min" in isolation. Always show both rate and count.
• Time range alignment — all panels must be aligned to the same time range. Misaligned ranges create false correlations between unrelated spikes.
• USE and RED on separate rows — infrastructure metrics and service metrics have different slopes, scales, and audiences.

Anti-Patterns to Avoid

Cardinality Management

Cardinality is the number of unique time-series a metric produces. It equals the product of unique values across all labels.

Rule: Label values must be bounded and low-cardinality (fewer than 100 unique values per label, ideally fewer than 20).

Detection: Monitor prometheus_tsdb_head_series — alert when the series count grows unexpectedly week-over-week. A sudden spike indicates a cardinality bug in newly deployed instrumentation.

Mitigation for existing high-cardinality metrics: Use metric_relabel_configs in the Prometheus scrape config to drop or hash the offending label before storage.

Metrics vs Logs vs Traces — Tradeoffs

The three pillars are complementary. A full observability stack uses all three: metrics to detect, traces to locate, logs to explain.

Related Concepts

• Micrometer — Java meter API (Counter, Timer, Gauge), Spring Boot auto-configuration, BFF-specific metric names
• Distributed-Tracing — request-scoped latency measurement via spans; complements metrics for per-request attribution
• Structured-Logging — event-level observability; the third pillar that explains why metrics changed
• SLO-SLI-SLA — metrics as the measurement basis for SLIs; error budgets derived from metric time-series
• Alerting-Strategies — turning metrics (burn rate, threshold, anomaly) into actionable on-call alerts

Backlinks

• Micrometer
• Distributed-Tracing
• SLO-SLI-SLA
• Alerting-Strategies
• Structured-Logging