16.1.3 Build Cache Surprises

Build cache surprises occur when Docker's layer caching reuses a previous build's result in a situation where a developer expected fresh execution, or conversely invalidates and re-executes a step the developer expected to be cached, and both directions of surprise stem from the same underlying mechanism: cache validity is determined by specific, sometimes non-obvious rules about what counts as a change to a given instruction.

How cache invalidation actually works

Docker's classic layer caching invalidates a layer, and every layer after it, when the instruction itself changes textually, or, for COPY and ADD, when the content of the files being copied changes, checked through a content hash rather than file modification time:

COPY package.json .
RUN npm install
COPY . .

This ordering, copying only the manifest file before running install, then copying the rest of the source afterward, is specifically structured so that source code changes (which only affect the final COPY . . layer) do not invalidate the considerably more expensive npm install layer, as long as package.json itself has not changed.

The surprise of an unexpectedly reused layer

A common surprise is a build that should reflect a recent dependency change but appears not to, because the file Docker is actually checking for changes was not the one that was modified:

COPY package.json package-lock.json .
RUN npm ci

If only package-lock.json was regenerated but the COPY instruction's cache key calculation considers both files together as a single layer input, a subtle bug in how the files are referenced, or an unexpected interaction with .dockerignore excluding one of them, can cause the layer to be reused incorrectly. Verifying directly with --no-cache is the most reliable way to confirm whether stale caching, rather than something else, explains an unexpected build result:

docker build --no-cache -t my-api .

The surprise of an unexpectedly invalidated layer

The opposite surprise, a layer that should have been cached and reused but was not, recurring full reinstallation of dependencies despite no apparent dependency change, often traces back to something appearing to change even when the actual dependency content did not:

COPY . .
RUN npm ci

Copying the entire source tree before running install means any change anywhere in the source, not just dependency-related files, invalidates this layer and forces a full reinstall on every build; restructuring to copy only the manifest and lockfile first, as in the earlier example, isolates the cache key to only what actually affects the install step.

ARG and ENV affecting cache keys

Build arguments and environment variables used within an instruction become part of that instruction's cache key, which means a build argument that changes between builds, even one unrelated to what a specific instruction is actually doing, can unexpectedly invalidate a layer that uses it anywhere in its command:

ARG BUILD_TIMESTAMP
RUN echo "Built at ${BUILD_TIMESTAMP}" && npm ci

A build timestamp argument that changes on every single build invalidates this entire layer every time, defeating any caching for the npm ci step that was combined into the same instruction; separating concerns that genuinely need to vary on every build from those that should be cached avoids this kind of unintentional cache-busting.

Multi-stage build cache interactions

In a multi-stage build, each stage's caching is generally independent, but a later stage referencing an earlier one through COPY --from= depends on that earlier stage having actually produced a fresh result if anything relevant to it changed, which can produce a subtle surprise if the dependency between stages is not fully understood:

FROM node:20 AS build
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html

If the build stage's cache was reused (because nothing it directly depends on changed) but the final stage's COPY --from=build instruction itself is reported as re-executed for an unrelated reason, the actual content copied into the final image is still correct, since it reflects the build stage's cached, valid output; this distinction, a layer being "re-executed" textually versus the underlying content actually changing, is worth understanding to avoid unnecessary concern over cache behavior that is functioning correctly.

Cache mounts and persistent caches across builds

BuildKit's cache mount feature introduces a different kind of caching, a persistent directory available across separate build invocations, distinct from the standard layer caching mechanism, and confusing the two can lead to surprises about what is and is not actually being reused:

RUN --mount=type=cache,target=/root/.npm npm install

This cache mount persists package manager download caches across builds even when the layer itself is invalidated and re-executed, which is a deliberate, separate caching mechanism specifically designed to speed up re-execution of an otherwise-invalidated step, not a replacement for understanding standard layer caching behavior.

Cache invalidation cascading to every subsequent instruction

A frequently surprising consequence of Docker's layer caching model is that invalidating any single layer invalidates every layer that follows it in the same stage, regardless of whether those later instructions would have produced an identical result on their own:

COPY . .
RUN npm ci
RUN npm run build
RUN npm run lint

A change to any file copied by the first instruction invalidates all three subsequent RUN instructions, even if, for instance, the lint step's behavior is entirely deterministic and would have produced the identical result regardless of what changed; this is simply how the caching model works, and structuring instructions to minimize unnecessary cascading invalidation, ordering from least to most frequently changing, is the practical response rather than expecting more granular invalidation than the model actually provides.

Common mistakes

• Copying the entire source tree before a dependency installation step, causing any source change to invalidate and force a full reinstall unnecessarily.
• Including a frequently changing build argument or environment variable in the same instruction as an expensive, otherwise-cacheable step, unintentionally busting its cache on every build.
• Assuming --no-cache is needed to confirm a real change, rather than first checking whether the actual file or argument believed to have changed is genuinely what the relevant instruction's cache key depends on.
• Confusing BuildKit cache mounts with standard layer caching, leading to incorrect assumptions about what is and is not being reused between builds.
• Not accounting for cascading invalidation, where a single early change invalidates every subsequent instruction in the same stage regardless of whether each one individually would have produced the same result.

Build cache surprises are resolved by understanding precisely what determines a given instruction's cache key, file content hashes for COPY and ADD, the literal instruction text including any interpolated ARG or ENV values, and structuring a Dockerfile deliberately, ordering from least to most frequently changing, to align actual cache behavior with what a developer intuitively expects to be reused versus re-executed.