Secrets Management

Tags

Secrets Management

Secrets management is the discipline of storing, distributing, rotating, and revoking credentials so that no secret lives in source code, environment variables baked into images, or long-lived flat files — applications receive the minimum scope for the minimum duration.

Intent

Secrets management is the discipline of storing, distributing, rotating, and revoking credentials (API keys, database passwords, TLS certificates, signing keys) so that no secret lives in source code, environment variables baked into images, or long-lived flat files. The goal is least-privilege access to secrets — applications receive the minimum scope for the minimum duration.

When NOT to Use

  • Do not build a custom secrets store when a managed solution (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) exists — custom implementations routinely miss rotation, audit logging, and lease expiry.
  • Do not use dynamic secrets for short-lived CLI scripts where the lease overhead exceeds the script lifetime; static secrets with short TTLs are acceptable.
  • Do not model Vault agent sidecar pattern for single-process CLIs or batch jobs that do not restart — the sidecar lifecycle assumes a long-running process.

How It Works

Static vs Dynamic Secrets

Static secrets: a credential is generated once, stored in the secrets store, and rotated on a schedule. The application reads the credential at start-up or via a watch loop. Rotation means generating a new value and updating the store; the application must be notified (restart, reload, or watch-based hot swap).

Dynamic secrets: the secrets store generates a time-bounded credential on demand (e.g., Vault database secrets engine issues a Postgres user valid for 1 hour). No long-lived credential exists. The application requests a credential, uses it, and the secrets store revokes it at lease expiry. Dynamic secrets are the preferred pattern for database credentials, cloud IAM tokens, and PKI certificates.

Dual-Credential Rotation Pattern

The naive rotation approach — invalidate old credential, write new credential, restart application — creates a gap window where both credentials are invalid. The dual-credential rotation pattern eliminates this:

  1. Generate new credential (Credential B) while Credential A remains active.
  2. Write Credential B to the secrets store.
  3. Notify all application instances (rolling restart or watch-triggered reload).
  4. Verify all instances are using Credential B.
  5. Revoke Credential A.

This pattern requires the target system (database, external API) to support two concurrent active credentials. Most databases support two active users; most APIs support two active API keys.

sequenceDiagram
    participant SM as Secrets Manager
    participant App as Application Instances
    participant DB as Database

    Note over SM,DB: Phase 1 — Generate
    SM->>DB: Create Credential B (B active, A still active)
    SM->>SM: Write B to secrets store

    Note over SM,App: Phase 2 — Rotate
    SM->>App: Notify: reload credentials
    App->>SM: Fetch Credential B
    App->>DB: Connect with Credential B

    Note over SM,DB: Phase 3 — Revoke
    SM->>DB: Revoke Credential A

Vault Agent Sidecar Pattern

The Vault agent runs as a sidecar alongside the application process. It authenticates to Vault using the platform identity (Kubernetes service account, AWS IAM role) and writes the fetched secret to a tmpfs volume or memory-mapped file. The application reads credentials from the local file system — it never communicates with Vault directly. The agent handles lease renewal and writes updated credentials when leases approach expiry.

See Secrets-Management-diagram for the rotation topology diagram.

Scope boundary: For Spring Cloud Vault configuration syntax (bootstrap.yaml, spring.cloud.vault.* properties), see the Spring Cloud Vault 4.x documentation. This note covers the pattern, not the configuration.

TypeScript Example

Rotation-aware secret fetch using AWS SDK v2 (@aws-sdk/client-secrets-manager). On authentication failure, re-fetches the secret before retrying — handles the credential rotation window.

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
 
const client = new SecretsManagerClient({ region: 'us-east-1' });
 
async function getSecret(secretId: string): Promise<string> {
  const response = await client.send(
    new GetSecretValueCommand({ SecretId: secretId })
  );
  return response.SecretString ?? '';
}
 
// Rotation-aware: on auth failure, re-fetch before retrying
async function withRotation<T>(
  secretId: string,
  fn: (secret: string) => Promise<T>
): Promise<T> {
  const secret = await getSecret(secretId);
  try {
    return await fn(secret);
  } catch (err: unknown) {
    if (isAuthError(err)) {
      const rotated = await getSecret(secretId);
      return fn(rotated);
    }
    throw err;
  }
}
 
function isAuthError(err: unknown): boolean {
  return err instanceof Error && err.message.includes('authentication');
}

Java Example

Spring Cloud Vault 4.x — secrets injected via @Value. The Vault agent renews leases; the application reads from environment. Use @RefreshScope for rotation-aware hot reload without restart.

// Spring Cloud Vault 4.x — secrets injected via @Value
// Vault agent renews leases; application reads from environment
@Service
public class DatabaseService {
 
    private final String dbPassword;
 
    public DatabaseService(@Value("${database.password}") String dbPassword) {
        this.dbPassword = dbPassword;
    }
 
    // For rotation-aware code: refresh context or use @RefreshScope
    // Spring Cloud Vault 4.x auto-renews leases without application restart
}
  • Encryption — envelope encryption for secrets at rest (DEK+KEK pattern)
  • Zero-Trust-Architecture — identity plane drives secrets access policies
  • API-Key-Authentication — API keys are a category of secret requiring lifecycle management
  • JWT — signing keys are secrets requiring rotation

(Leave as placeholder — backlinks added in Phase 40 sweep)

Linked mentions