9.10 Kubernetes Container Restart Behavior

Kubernetes Container Restart Behavior is the set of rules and mechanisms the kubelet applies when deciding whether, when, and how quickly to restart a container after it has terminated, whether due to a normal exit, a crash, a failed liveness probe, or an out-of-memory kill. This behavior is governed primarily by the Pod's restartPolicy field combined with an exponential backoff mechanism designed to prevent unstable containers from consuming excessive node resources through rapid, repeated restart attempts.

The restartPolicy Field

Always

Under restartPolicy: Always, the kubelet restarts a container every time it terminates, regardless of whether the exit was successful or a failure. This is the default policy and is appropriate for long-running services that are expected to run indefinitely, since any termination is treated as an unexpected condition warranting a restart.

OnFailure

Under restartPolicy: OnFailure, the kubelet restarts a container only if it terminates with a non-zero exit code, treating a clean exit as an intentional completion that should not trigger a restart. This policy is commonly used for Pods managed by Jobs, where successful completion should end the container's lifecycle rather than restart it.

Never

Under restartPolicy: Never, the kubelet does not restart a container under any circumstances once it has terminated, leaving it in the Terminated state permanently. This is useful for one-shot tasks where any restart attempt, successful or not, would be inappropriate or redundant.

Exponential Backoff Mechanism

Growing Delay Between Attempts

When a container is eligible for restart, the kubelet does not restart it instantly on every failure; instead, it applies a backoff delay that starts at a short duration and doubles with each consecutive failure, up to a capped maximum interval, typically five minutes, after which the delay stops increasing further.

Backoff Reset Conditions

The accumulated backoff delay resets once a container has run successfully for a sufficient duration, generally interpreted as remaining in the Running state for around ten minutes without failing again, at which point a subsequent failure begins the backoff calculation from its initial short delay rather than continuing from the previously escalated value.

CrashLoopBackOff as an Observable Symptom

The CrashLoopBackOff reason reported in a container's Waiting state is the visible manifestation of this backoff mechanism, indicating that the container has failed and restarted multiple times in succession and is currently waiting out its backoff delay before the next attempt, rather than being stuck or unresponsive.

Restart Count Tracking

restartCount Field

Each container status includes a restartCount field that increments by one every time the kubelet restarts that specific container, providing a persistent, cumulative counter that survives across many restart cycles and serves as one of the most direct indicators of an unstable workload when observed over time.

lastState Preservation

Alongside the restart count, the lastState field retains the exit code, reason, and timestamps from the container's most recent termination prior to its current run, ensuring that diagnostic information about the last failure remains accessible even after a successful restart has moved the container back into the Running state.

Interaction With Liveness Probes

Probe-Triggered Restarts

A failing liveness probe that exceeds its configured failureThreshold results in the kubelet killing the container, which then becomes subject to the same restart policy and backoff behavior as any other termination, meaning probe-triggered restarts are not a separate mechanism but rather one of several possible triggers feeding into the same underlying restart logic.

No Backoff Bypass for Probe Failures

Repeated liveness probe failures do not bypass or accelerate past the exponential backoff schedule; a container failing its liveness probe in a tight loop will still experience increasing delays between restart attempts, exactly as it would for a container crashing on its own without any probe configured.

Pod-Level Versus Container-Level Scope

Restart Policy Applies Pod-Wide

The restartPolicy field is set once at the Pod level and applies uniformly to every container within that Pod, meaning individual containers cannot be configured with different restart policies from one another, even though their actual restart histories and backoff timers are tracked independently per container.