17.1.2 Reproducible Image Practices

Reproducible image practices ensure that building the same Dockerfile against the same source code produces a bitwise-identical, or at least functionally identical, image every time, regardless of when or where that build runs, a property that matters for security verification, debugging confidence, and trusting that what was tested is genuinely and exactly what gets deployed.

Why reproducibility is not automatic

A naive Dockerfile can produce a different image on every build even with no source code change at all, since several common practices introduce non-determinism: pulling a mutable base image tag that has since been updated, installing a dependency version range that resolves differently as new versions are published, or embedding a build timestamp directly into the image:

FROM node:20
RUN npm install express

Both the base image and the dependency version here can resolve to different actual content on two builds performed days apart, even though the Dockerfile itself never changed at all, which means two builds from the identical source can produce genuinely different images without anyone having made a deliberate change.

Pinning base images by digest

Referencing a base image by its specific content digest, rather than a mutable tag, removes the most significant source of build-to-build variation, since a digest always refers to exactly the same, immutable content regardless of when the build runs:

FROM node:20@sha256:3f29a8c1d8e2b4f6a9c7d5e8b1f3a6c9d2e5f8b1a4c7d0e3f6a9c2d5e8b1f4a7

This trades automatic upstream updates for deliberate, explicit control over exactly when the base image actually changes, requiring a conscious decision to bump the digest reference rather than silently picking up whatever the tag happens to currently point to.

Pinning dependency versions exactly

Lockfiles, when actually present, committed, and respected by a strict installation command, ensure dependency resolution produces the identical version set on every build, rather than re-resolving against whatever the latest compatible versions happen to be at build time:

npm ci

pip install -r requirements.txt --no-deps

pip-compile requirements.in

npm ci specifically requires and respects the lockfile exactly, failing rather than silently re-resolving if the lockfile and manifest are out of sync, which is the behavior reproducibility depends on; a more permissive install command that re-resolves dependencies on every invocation undermines this guarantee even with a lockfile present.

Avoiding embedded timestamps and non-deterministic metadata

Build instructions that embed the current time, a random value, or any other non-deterministic data directly into the image's content produce a different result on every build by design, which is occasionally intentional but should be a deliberate, isolated exception rather than something that accidentally affects the bulk of the image's actual functional content:

RUN echo "Built at $(date)" > /app/build-info.txt

If build provenance information like this is genuinely needed, isolating it to a clearly separate, small layer or metadata label, rather than letting it influence layer ordering or caching for the application's actual functional content, keeps the bulk of the image reproducible while still capturing the build timestamp somewhere accessible:

LABEL org.opencontainers.image.created="2024-06-01T12:00:00Z"

A build-time label is metadata about the image rather than content within it, which is a meaningfully different, more contained way to capture this kind of information without affecting the reproducibility of the image's actual filesystem content.

File ordering and timestamp non-determinism within layers

Some build tools and archive creation processes can introduce subtle non-determinism through file ordering or embedded timestamps within a layer's own tar archive, even when the actual file content is otherwise identical between two builds, which is a more advanced, lower-level concern relevant specifically for projects pursuing strict, bit-for-bit reproducibility rather than just functional consistency:

docker buildx build --provenance=false --sbom=false -t my-api .

For most projects, this level of strict, bit-for-bit reproducibility is not the actual goal; functional reproducibility, the same dependency versions and base image content producing equivalent, verifiably consistent behavior, is generally sufficient and considerably easier to achieve without needing to control for this deeper level of build tooling determinism.

Verifying reproducibility directly

Building the same source twice, ideally on different machines or at meaningfully different times, and comparing the results directly confirms whether reproducibility has actually been achieved rather than only assumed based on following the practices described above:

docker build -t my-api:test1 .
docker build -t my-api:test2 .
docker run --rm my-api:test1 npm ls --all > deps1.txt
docker run --rm my-api:test2 npm ls --all > deps2.txt
diff deps1.txt deps2.txt

A clean diff across the actual installed dependency tree, even if the images are not bit-for-bit identical at the layer level, provides strong, practical confidence that the build is functionally reproducible in the way that actually matters for most purposes.

Reproducibility as a security and incident response asset

Beyond general engineering hygiene, reproducible builds provide a concrete benefit during a security investigation: confirming that a deployed image's content can be regenerated and verified against its claimed source, rather than relying entirely on trust in the build pipeline's own historical record, which is considerably more valuable when investigating a potential supply chain compromise specifically:

git checkout v1.4.2
docker build -t my-api:verify .
docker inspect my-api:verify --format '{{.Id}}'
docker inspect registry.example.com/my-api:1.4.2 --format '{{.Id}}'

Comparing a freshly rebuilt image's content against what is actually running in production, from the exact same tagged source commit, is a meaningful, concrete verification step available specifically because the build process was deliberately made reproducible.

Common mistakes

• Referencing base images by mutable tag rather than digest, allowing the same Dockerfile to silently produce different content depending purely on when the build happened to run.
• Relying on a permissive dependency installation command that re-resolves versions on every build rather than a strict, lockfile-respecting one.
• Embedding build timestamps or other non-deterministic values directly into the image's functional content rather than isolating them to metadata labels.
• Pursuing strict, bit-for-bit reproducibility when functional reproducibility, verified through dependency tree comparison, would be sufficient and considerably easier to achieve.
• Never actually testing reproducibility directly by building the same source twice and comparing results, relying instead on assumption that following best practices alone guarantees the property.

Reproducible image practices, digest-pinned base images, strict lockfile-respecting dependency installation, and isolated build metadata, together ensure that the same source genuinely produces the same, verifiable result on every build, which provides concrete value for debugging confidence, security verification, and incident response that a non-reproducible build pipeline cannot offer.