Kubernetes Secret Safety Guidelines

Kubernetes Secret Safety Guidelines describe the practices for storing, distributing, and rotating sensitive values — credentials, API keys, TLS private keys, tokens — using the Secret resource, given that Kubernetes' built-in Secret type provides considerably weaker protection by default than its name implies, and genuine safety requires deliberate, layered controls beyond simply choosing Secret over ConfigMap.

What Secret Actually Protects Against

Base64 Is Encoding, Not Encryption

Secret data is stored base64-encoded, which is trivially reversible and provides no confidentiality on its own — it exists to allow arbitrary binary data in a JSON/YAML-based API, not to protect the value from anyone who can read the object. Treating base64 encoding as security is a common and dangerous misunderstanding.

Encryption at Rest Must Be Explicitly Enabled

By default, Secrets are stored in etcd without encryption at rest, meaning anyone with access to etcd's underlying storage or backups can read every Secret in plaintext. Enabling encryption at rest (via an EncryptionConfiguration with a KMS or local key provider) is a foundational, non-optional step for any cluster actually storing sensitive data in Secrets.

RBAC Is the Primary Access Control

The meaningful protection Secrets provide over ConfigMaps comes from RBAC treating secrets as a distinct, more tightly restricted resource type by convention — but that protection only holds if RBAC is actually configured that way. A Role granting broad get/list on Secrets across a namespace defeats the purpose regardless of the underlying storage encryption.

Preferring External Secret Management

Kubernetes Secrets as a Delivery Mechanism, Not a Source of Truth

The safest pattern treats Kubernetes Secret objects as a synced delivery mechanism rather than the authoritative store — the actual secret lives in a dedicated secret management system (a cloud KMS-backed secrets manager, Vault, or equivalent), and an operator (such as External Secrets Operator) syncs values into Kubernetes Secrets on demand, keeping the system of record, audit trail, and rotation logic outside etcd entirely.

CSI Secret Store Integration

The Secrets Store CSI Driver mounts secrets directly from an external provider into a Pod's filesystem without ever creating a corresponding Kubernetes Secret object at all, removing the secret from etcd's storage and from the Kubernetes API's exposure surface entirely — appropriate for the most sensitive values where even a properly RBAC'd Secret object is considered too much exposure.

Rotation

Short-Lived Credentials Where Possible

Wherever the backing system supports it, dynamically generated, short-lived credentials (a database password issued for a bounded lease rather than a static long-lived one) reduce the value of a leaked credential dramatically, since it expires on its own regardless of whether the leak is ever detected.

Coordinating Rotation With Consuming Workloads

Rotating a Secret's value in Kubernetes does not automatically propagate to running Pods consuming it via environment variables, mirroring the same staleness behavior as ConfigMaps — a rotation process must also trigger a rollout of consuming workloads (via the same checksum-annotation pattern used for ConfigMaps) or ensure the application re-reads a mounted volume secret rather than caching the value it read at startup.

Reducing Exposure Surface

Minimizing Secret Scope and Lifetime

Secrets should be scoped to the smallest set of workloads that actually need them, in the namespace where those workloads run — a Secret needed by only one Deployment shouldn't be placed somewhere broadly readable across a namespace shared by many unrelated workloads.

Avoiding Secrets in Version Control

Raw Secret manifests must never be committed to version control, even in a private repository, since access boundaries and history retention for source control differ from those intended for sensitive runtime values. Sealed Secrets, SOPS-encrypted files, or GitOps patterns that keep only encrypted material in git are the appropriate way to manage Secret definitions declaratively.

Preventing Accidental Logging

Application code and infrastructure logging should be audited to ensure secret values are never accidentally written to logs, error messages, or crash dumps — a leaked value that's been quietly persisted into a log aggregation system for months is functionally equivalent to a permanent compromise of that credential.

Operational Safeguards

Audit Access to Secrets Specifically

API server audit logging should, at minimum, capture read access to Secret objects, since unauthorized access to a Secret is itself a security-relevant event independent of whatever the credential is later used for.

Immutable Secrets Where Values Don't Change

As with ConfigMaps, immutable: true prevents unintended runtime edits to a Secret's data and improves kubelet watch performance, appropriate for Secrets whose values are only ever replaced by a full rotation process rather than incrementally patched.

Example Configuration

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: codartium-db-credentials
  namespace: codartium
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: cloud-secret-store
    kind: ClusterSecretStore
  target:
    name: codartium-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/codartium/db-password

Practical Consequences

Following these guidelines produces a system where sensitive values are encrypted at rest, tightly scoped by RBAC, rotated on a schedule with limited blast radius per leak, and never persisted in a form that outlives their intended lifetime. Neglecting them commonly results in credentials sitting in plaintext in an unencrypted etcd backup, secret values discoverable by anyone with broad namespace read access, or a credential leaked years earlier still valid today because no rotation process was ever established to invalidate it.