4.8 Kubernetes Image Runtime Handling

Kubernetes Image Runtime Handling is the set of operations performed by the container runtime, under kubelet direction, to resolve, retrieve, store, and manage the container images required to run pods on a node. It spans authentication with remote registries, layer-level caching, local disk management, and garbage collection, all mediated through the ImageService portion of the Container Runtime Interface.

The ImageService Interface

Separation from RuntimeService

The Container Runtime Interface splits container runtime responsibilities into two gRPC services: RuntimeService, which manages sandboxes and containers, and ImageService, which manages images independently. This separation allows image operations, such as pulling or listing, to proceed without requiring an active container or sandbox context.

Core ImageService Operations

ImageService exposes operations including PullImage, ListImages, ImageStatus, and RemoveImage. The kubelet calls these methods when preparing to create a container, when reporting node image inventory, and when performing garbage collection to reclaim disk space.

Image Reference Resolution

Registry, Repository, and Tag Structure

An image reference typically consists of a registry hostname, a repository path, and either a tag or a content-addressable digest, such as registry.example.com/team/app:1.4.2 or the same reference pinned to a sha256 digest for immutability.

Default Registry Assumptions

When an image reference omits a registry hostname, the runtime applies a configured default registry, commonly a public registry such as Docker Hub, according to the runtime's own configuration rather than any Kubernetes-specific default, since image resolution happens entirely within the runtime layer.

Digest Pinning for Reproducibility

Referencing images by digest rather than by mutable tag ensures that the exact same image content is pulled every time, which is important for reproducible deployments and for avoiding unexpected behavior changes when a tag like latest is updated upstream.

Pull Policy and Caching Behavior

ImagePullPolicy Semantics

The kubelet enforces three pull policy values. Always forces a pull attempt on every container creation, verifying the image digest even if a local copy exists. IfNotPresent skips pulling when a matching image is already cached locally. Never prevents any pull attempt, requiring the image to already exist on the node or the container creation fails.

Default Policy Inference

When no explicit imagePullPolicy is set, the kubelet infers a default based on the image tag: images tagged latest default to Always, while images with any other explicit tag default to IfNotPresent, reflecting the differing mutability expectations of these two tagging conventions.

Local Image Cache

Pulled image layers are stored in a local content store managed by the runtime, commonly backed by overlayfs. Subsequent pulls of images sharing layers with already-cached images only need to fetch the missing layers, reducing network transfer and pull latency.

Authentication and Private Registries

Image Pull Secrets

For private registries, the kubelet retrieves credentials from imagePullSecrets referenced in the pod spec or attached to the pod's service account, and passes them to the runtime's PullImage call so that authentication can occur against the target registry.

Credential Providers

Cloud-managed Kubernetes environments often use dynamic credential provider plugins that the kubelet invokes to obtain short-lived registry credentials, such as temporary tokens for a cloud provider's container registry, avoiding the need to store long-lived static credentials in the cluster.

Disk Usage and Garbage Collection

Image Garbage Collection Thresholds

The kubelet monitors disk usage consumed by container images and triggers garbage collection when usage crosses a configured high-water-mark threshold, removing unused images, starting with the least recently used, until usage falls below a configured low-water-mark threshold.

Unused Image Detection

An image is considered a garbage collection candidate when no running or recently stopped container references it. The kubelet consults the runtime's image and container inventories to determine which images are currently unreferenced and therefore safe to remove.

Node Reported Image Inventory

As part of node status reporting, the kubelet includes a list of images already present on the node, along with their reported sizes. The scheduler can use this information as a soft signal when making placement decisions, since scheduling a pod onto a node that already has its image cached can reduce pod startup latency.

Multi-Architecture and Manifest Lists

Manifest List Resolution

Many images are published as manifest lists, sometimes called multi-arch images, containing references to platform-specific image variants for architectures such as amd64 and arm64. The runtime resolves the manifest list against the node's actual architecture and operating system to select the correct underlying image variant to pull.

Platform Mismatch Failures

If a manifest list does not include a variant matching the node's platform, the pull operation fails, and the kubelet reports the resulting error as part of the container's waiting state, commonly surfaced as an image pull error in pod events.