secrets-management-expert

Secrets Management Expert

A secret in source control is compromised the moment it is pushed, and it stays compromised after deletion because history is forever. Everything below follows from that.

Core Concepts

The Hierarchy

Ranked by how much damage a compromise causes:

1. No secret at all — workload identity. The platform proves who the workload is; no credential exists to steal. The target state wherever it is available.
2. Dynamic, short-lived credentials — minted per session, expiring in minutes. Theft has a short window.
3. Static secret in a manager, rotated on a schedule, accessed by authenticated identity, audited.
4. Static secret in an environment variable, injected at deploy.
5. Secret in a config file on disk.
6. Secret in source control. Compromised.

Move each secret up this list rather than aiming for perfection everywhere.

Envelope Encryption

Encrypt data with a data key; encrypt the data key with a key that never leaves the KMS. Store the wrapped data key beside the ciphertext.

plaintext --[AES-256-GCM, data key]--> ciphertext
data key  --[KMS master key]---------> wrapped data key   (stored together)

This is what lets you rotate the master key without re-encrypting terabytes, and what keeps the master key material inside a boundary you can audit.

Rotation Only Works If It Is Routine

A rotation procedure exercised once a year fails when you need it during an incident. Rotate on a schedule so the mechanism is known to work — that, not the interval, is the point.

Getting Secrets Out of Code

import os

# Wrong: a default makes a misconfiguration silently insecure
API_KEY = os.environ.get("API_KEY", "dev-key-123")

# Right: fail closed, at startup, with a clear message
def required(name: str) -> str:
    try:
        return os.environ[name]
    except KeyError:
        raise RuntimeError(f"{name} is required and not set") from None

API_KEY = required("API_KEY")

Validate every required secret at startup, not at first use. A service that boots successfully and fails at 3 a.m. on the first payment is worse than one that refuses to start.