17.1.3.4 Secretless Image Layers

Secretless image layers means ensuring that no layer in an image's entire history, not just its final, visible filesystem state, ever contains sensitive material, which requires understanding precisely how layer persistence works at a mechanical level, since a secret "removed" in a later instruction remains fully recoverable from the earlier layer that originally introduced it.

Why removal in a later layer does not help

Each Dockerfile instruction that modifies the filesystem creates a new, immutable layer, and a later instruction that deletes a file only adds a new layer recording that deletion; the earlier layer containing the original file remains part of the image's history and is still fully present and extractable:

COPY credentials.json /tmp/credentials.json
RUN cat /tmp/credentials.json && rm /tmp/credentials.json

docker save my-api -o my-api.tar
tar -xf my-api.tar

Extracting the saved image's raw layers directly reveals the original credentials.json file sitting in the layer created by the COPY instruction, completely unaffected by the later RUN instruction's deletion, since that deletion only exists as a separate, later layer rather than retroactively modifying the earlier one.

Auditing existing images for layer-level secret exposure

Tools designed specifically to inspect every layer of an image, rather than just its final, merged filesystem state, can surface secrets that would otherwise remain hidden from a simple inspection of the running container's current filesystem:

dive my-api:1.4.2

trufflehog docker --image=my-api:1.4.2

Running a dedicated secret-scanning tool specifically against an image's full layer history, rather than only against its source repository or its final, running filesystem state, catches exactly this class of exposure, a secret that was present at some point during the build and was later deleted but never actually removed from the image's persisted history.

Build secrets through BuildKit's dedicated mechanism

BuildKit's --mount=type=secret provides the only mechanism that genuinely avoids this problem for a secret needed temporarily during a specific build step, since the secret is made available only for the duration of that specific RUN instruction and is never written into any layer at all, persisted or otherwise:

RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm install

docker build --secret id=npm_token,src=./npm_token.txt -t my-api .

This is mechanically different from copying a secret file and deleting it afterward; the secret mount's content never becomes part of any layer's filesystem snapshot in the first place, which is the only approach that genuinely guarantees the secret cannot be recovered from the image's layer history afterward.

Why squashing is not a reliable substitute

Some tooling supports squashing an image's layers into a single, combined layer after the build completes, which can superficially appear to address this concern by collapsing the visible history into one layer; however, this is not a reliable, guaranteed substitute for never introducing the secret in the first place, since squashing behavior and what it actually preserves or discards can vary by tool and version, and relying on it as the primary safeguard is considerably less certain than simply never having the secret enter any layer at all:

docker build --squash -t my-api .

Treating squashing as a defense-in-depth, secondary measure rather than the primary control is the more reliable posture; the BuildKit secret mount mechanism, which structurally prevents the secret from entering any layer at all, remains the actual, dependable fix.

Avoiding secrets in build arguments

Build arguments (ARG) are also recorded in image metadata and history, which means passing a secret as a build argument has the same fundamental persistence problem as copying it into a file, despite feeling like a different, perhaps safer mechanism:

ARG API_KEY
RUN curl -H "Authorization: Bearer $API_KEY" https://api.example.com/setup

docker history my-api --no-trunc | grep API_KEY

Checking an image's history directly after using a build argument this way frequently reveals the actual secret value recorded as part of the instruction's own history entry, confirming this approach provides no real protection despite superficially seeming more contained than a copied file.

Verifying secretless layers as a routine build pipeline check

Incorporating an automated check for this specific class of issue directly into the CI pipeline, rather than relying on manual review, catches an accidental secret-in-layer mistake before the image is ever pushed to a registry where it would become more widely accessible:

verify-no-secrets:
  script:
    - trufflehog docker --image=my-api:$CI_COMMIT_TAG --fail

Configuring this check to actually fail the pipeline on a detected finding, rather than only reporting it informationally, makes secretless layer verification an enforced gate rather than an optional, easily ignored advisory step.

Rotating any secret discovered to have leaked this way

If a secret is discovered to have been present in an already-pushed image's layer history, the correct response is treating it as compromised and rotating it immediately, the same as any other confirmed credential leak, rather than only removing it from future builds going forward while leaving the original, leaked value still valid:

vault write database/rotate-root/my-postgres-db

Any already-pushed image containing the leaked secret remains a recoverable source of that exact value for as long as it exists in any registry or any host that has previously pulled it, which is precisely why rotation, not just future prevention, is the necessary, complete response once a leak through this mechanism is confirmed.

Common mistakes

• Assuming a secret deleted in a later instruction is genuinely removed from the image, when the earlier layer containing it remains fully present and recoverable.
• Auditing only an image's current, merged filesystem state for secrets, missing exposure that exists specifically within an earlier, since-modified layer's history.
• Treating image squashing as a reliable, primary mechanism for removing a secret from layer history rather than as, at best, a secondary, defense-in-depth measure.
• Passing a secret as a build argument under the mistaken impression that this is meaningfully safer than copying it into a file, when both are recorded in persistent image history.
• Discovering a leaked secret in an already-pushed image's layer history and only removing it from future builds without also rotating the actual, already-exposed credential value.

Secretless image layers require understanding that layer persistence is permanent and additive, never retroactively modified by a later instruction, which makes BuildKit's dedicated secret mount mechanism the only genuinely reliable way to use a secret during a build without it ever becoming part of any layer, and makes layer-aware auditing tools, rather than only source repository or running-container scans, necessary to catch this specific class of exposure in images that may have already been built using a less reliable approach.