17.2.2.4 Secret Manager Practice

Secret manager practice addresses the adoption decision and maturity progression around secrets management, distinct from the mechanical details of how a secrets manager actually integrates with a container, focused on when adopting one is genuinely warranted, how to migrate toward one incrementally, and the ongoing organizational habits, access review, rotation cadence, that determine whether the investment actually pays off in practice.

Starting simple and graduating deliberately

A small project with one or two services and a handful of secrets can reasonably start with mounted secret files or Docker's native secrets mechanism, which provide meaningful protection over plain environment variables without the operational overhead of standing up and maintaining a full, dedicated secrets manager:

services:
  api:
    secrets:
      - db_password

As the number of services, environments, and people needing some level of access grows, the operational case for a dedicated secrets manager, centralized rotation, fine-grained access policy, audit logging, strengthens; recognizing the point at which this graduation genuinely makes sense, rather than either adopting a full secrets manager prematurely or delaying it well past the point of genuine need, is the actual judgment call this practice is about.

Signals that indicate it is time to adopt a dedicated secrets manager

Several concrete signals suggest mounted files or native Compose secrets are no longer sufficient: more than a handful of people need some level of credential access, credentials need to rotate more frequently than manual file updates can reasonably keep pace with, or compliance requirements demand a documented audit trail of exactly who accessed which secret and when:

ls /run/secrets/

If managing this growing set of secrets has become a recurring, manual coordination burden, repeatedly distributing updated files to the right machines, tracking who currently has access to what, this operational friction itself is a strong, practical signal that the investment in a dedicated secrets manager has likely become worthwhile.

Migrating incrementally rather than all at once

A full migration from plain environment variables or mounted files to a dedicated secrets manager does not need to happen for every service simultaneously; migrating the most sensitive, highest-risk credentials first, while leaving lower-risk configuration on its existing mechanism temporarily, delivers the most important security benefit earliest while spreading the migration effort over a more manageable timeline:

const dbPassword = process.env.USE_VAULT === 'true'
  ? await fetchFromVault('secret/db-password')
  : fs.readFileSync('/run/secrets/db_password', 'utf8');

A feature-flagged transition period like this, while not elegant as a permanent state, allows verifying the new secrets manager integration works correctly for one credential before committing to migrating everything simultaneously.

Establishing least-privilege access policy as routine practice

Beyond the technical integration, the actual security benefit of a secrets manager depends on access policies being genuinely scoped narrowly, each service or person granted access only to the specific secrets they actually need, reviewed and maintained as an ongoing practice rather than configured once and left unexamined indefinitely:

path "secret/data/my-api/*" {
  capabilities = ["read"]
}

A policy granting broad, unscoped access "to be safe" or "for convenience" undermines much of the actual benefit the secrets manager was adopted to provide, since a single compromised service's credentials would then be able to read every other service's secrets as well.

Scheduling regular access reviews

Access granted at one point in time, when a specific need genuinely existed, can become stale as that need changes or disappears entirely; scheduling a regular, routine review of who and what currently has access to which secrets catches accumulated, no-longer-necessary access before it becomes a meaningful, unaddressed risk:

vault read sys/policies/acl/my-api-policy

0 9 1 * * /usr/local/sbin/generate-access-review-report.sh

A monthly or quarterly scheduled review, even if largely confirming that existing access remains appropriate, is what keeps access genuinely aligned with current, actual need rather than reflecting a snapshot of needs from whenever access was originally granted.

Establishing a rotation cadence and rehearsing it

Adopting a secrets manager makes rotation mechanically easier, but the actual discipline of rotating on a defined, regular schedule, and periodically rehearsing the rotation process itself to confirm it still works correctly as the system evolves, is a separate, ongoing organizational practice that the tooling alone does not guarantee:

vault write database/rotate-root/my-postgres-db

./scripts/rotate-credential.sh --dry-run db_password

A rotation mechanism that exists but has never actually been exercised carries real risk of failing, or producing unexpected disruption, the first time it is genuinely needed, which is the same reasoning that applies to backup restore testing generally.

Measuring whether the investment is actually paying off

Periodically assessing whether the adopted secrets manager is genuinely reducing risk and operational burden, fewer manual credential distribution incidents, faster incident response when a credential needs emergency rotation, a clean audit trail actually used during a security review, confirms the original adoption decision continues to be justified, rather than assuming it indefinitely based on the initial decision alone.

Common mistakes

• Adopting a full, dedicated secrets manager prematurely for a small project where mounted files or native Compose secrets would have been entirely sufficient.
• Delaying secrets manager adoption well past the point where managing an increasingly complex set of credentials manually has become a genuine, recurring operational burden.
• Migrating every credential simultaneously rather than starting with the highest-risk ones and proceeding incrementally.
• Granting broad, unscoped access policies for convenience, undermining much of the actual security benefit the secrets manager was adopted to provide.
• Never scheduling regular access reviews or rehearsing the rotation process, leaving both to atrophy or fail silently until an actual incident forces the question.

Secret manager practice is fundamentally a judgment call about timing and an ongoing organizational discipline, recognizing the concrete signals that indicate a dedicated secrets manager has become genuinely warranted, migrating to it incrementally starting with the highest-risk credentials, and maintaining least-privilege access policy, regular access review, and rehearsed rotation as routine practice rather than one-time setup tasks.