17.1.3.5 Base Refresh Practice

Base refresh practice covers the operational mechanics of actually carrying out scheduled base image updates safely, going beyond the general principle that base images should be periodically refreshed, into the specific process, testing gates, rollout sequencing, and rollback planning, that prevents a routine refresh from introducing an unexpected regression into production.

Treating a base refresh as a real change, not a no-op

Even though a base image refresh often involves no application code change at all, it can still introduce a meaningful behavioral difference, a patched library with a subtly different default behavior, a changed default configuration value, an updated system tool with different output formatting that a script depends on parsing; treating the refresh as a genuine, testable change rather than an automatic, risk-free no-op is the foundation of doing this safely:

FROM node:20.11.0

FROM node:20.11.1

This kind of patch-level base image bump looks trivial, and usually is, but "usually" is not the same as "always," which is why the same testing rigor applied to an application code change should also apply here, proportional to the actual risk, rather than skipped entirely on the assumption that a base image patch update is inherently safe.

Automating the refresh proposal, not the merge

Tools like Renovate or Dependabot can automatically detect an available base image update and open a pull request proposing it, which is valuable for surfacing the opportunity reliably and on schedule, but the actual merge should still pass through the same review and testing gate as any other change rather than being automatically merged without verification:

{
  "datasource": "docker",
  "depName": "node",
  "currentDigest": "sha256:3f29a8c1...",
  "newDigest": "sha256:8a1f3c9b..."
}

automerge: false

Disabling automerge specifically for base image update pull requests, even while allowing automation for the proposal and even the initial build and test run, keeps a human decision point in the loop before the change actually reaches production.

Running the full test suite against the refreshed base

The pull request proposing a base image update should trigger the exact same build and test pipeline as any other code change, building the application against the new base and running the full test suite before the update is considered ready to merge:

test-base-refresh:
  script:
    - docker build -t my-api:base-refresh-test .
    - docker run --rm my-api:base-refresh-test npm test

A test failure specifically correlated with the base image change, rather than any application code change, is exactly the kind of regression this process is designed to catch before it ever reaches production, and catching it here, in a pull request, is considerably less costly than catching it after deployment.

Canary or staged rollout for the refreshed image

Even after passing automated tests, rolling out a refreshed base image gradually, to a small subset of traffic or instances first, rather than to the entire production fleet simultaneously, provides an additional safety margin against a subtle issue that automated tests did not happen to catch:

services:
  api-canary:
    image: registry.example.com/my-api:base-refreshed
    deploy:
      replicas: 1
  api-stable:
    image: registry.example.com/my-api:current
    deploy:
      replicas: 9

Monitoring the canary instance's error rate and key metrics for a defined observation period before proceeding to a full rollout is the same canary deployment pattern applied here specifically to a base image refresh rather than an application code release.

Planning the rollback path before rolling out

Because a base refresh can introduce a regression not caught by automated testing, having a clear, fast rollback path, reverting to the previous, known-good base image digest, ready before the refresh is rolled out broadly, rather than figured out reactively during an actual incident, reduces the impact of any issue that does surface:

docker pull registry.example.com/my-api@sha256:3f29a8c1...
docker compose up -d

Keeping the previous image readily available and the rollback command rehearsed, the same discipline applied to any production deployment generally, applies equally here.

Batching versus staggering refreshes across multiple services

For an organization with many services sharing the same or similar base images, deciding whether to refresh them all simultaneously or stagger the rollout across services is a deliberate choice with different risk profiles: simultaneous refresh surfaces a systemic issue quickly but risks broad, simultaneous impact if one exists; staggered refresh limits blast radius per service but takes longer to fully roll out the update everywhere:

for service in api worker scheduler; do
  docker build --pull -t "$service:base-refreshed" "./$service"
done

A reasonable middle ground refreshes one representative, lower-risk service first as an informal canary for the base image update itself, then proceeds to the remaining services once that first refresh has run successfully in production for a defined observation period.

Documenting what changed in the base

Reviewing and recording what specifically changed in the new base image version, the upstream changelog, security advisories addressed, any documented breaking changes, provides context for both the review process and any future investigation if an issue is later traced back to this specific refresh:

docker buildx imagetools inspect node:20.11.1 --format '{{ json . }}'

A brief note in the pull request itself summarizing what the refresh actually addresses, rather than a bare digest or version bump with no context, makes the review meaningfully more informed than asking a reviewer to evaluate an opaque, unexplained change.

Common mistakes

• Treating a base image refresh as a risk-free, automatic no-op rather than a genuine change warranting the same testing rigor as any other.
• Allowing automated dependency update tooling to merge base image refresh pull requests without any human review or test verification gate.
• Rolling out a base refresh to an entire production fleet simultaneously without a canary or staged rollout step to catch a subtle, untested regression early.
• Not having a rehearsed, ready rollback path before rolling out a base refresh broadly, leaving rollback to be figured out reactively if an issue surfaces.
• Refreshing many services' shared base images simultaneously without a deliberate strategy balancing systemic issue detection against blast radius containment.

Base refresh practice treats the operational mechanics of carrying out a base image update with the same care as any other production change, automated proposal generation paired with human review, full test suite verification, staged or canary rollout, and a rehearsed rollback path, rather than assuming the update is inherently low-risk simply because it does not involve an application code change.