Kubernetes Pod Recovery Basics

Kubernetes Pod Recovery Basics is the set of mechanisms the kubelet applies directly to a single pod's containers to recover from failure without involving any higher-level controller, covering restart policy semantics, exponential backoff behavior, exit code and OOM signal interpretation, and the distinction between restarting an existing container and recreating a pod entirely.

restartPolicy Semantics

The Three Policy Values

spec:
  restartPolicy: Always

Always restarts a container on any exit, success or failure, appropriate for long-running services; OnFailure restarts only on a non-zero exit code, appropriate for batch workloads that should not be restarted after completing successfully; Never never restarts a container regardless of exit code, appropriate for one-shot jobs whose failure should be surfaced rather than retried by the kubelet directly.

restartPolicy Applies Per Pod, Not Per Container

restartPolicy is set once at the pod level and applies uniformly to every container within it; a pod cannot mix restart policies across its own containers, which is a design consideration when deciding whether a multi-container pod's sidecar and main application container genuinely belong in the same pod if their desired restart semantics actually differ.

Exponential Backoff and CrashLoopBackOff

The Backoff Algorithm

Restart 1: immediate
Restart 2: 10s delay
Restart 3: 20s delay
Restart 4: 40s delay
...
Restart n: min(10s x 2^(n-1), 300s)

After the first restart, the kubelet applies an exponentially increasing delay before each subsequent restart attempt, capped at five minutes, specifically to avoid a persistently crashing container consuming unbounded CPU and log volume through an unthrottled restart loop.

Recognizing and Diagnosing CrashLoopBackOff

kubectl get pods
NAME        READY   STATUS             RESTARTS
web-abc123  0/1     CrashLoopBackOff   7

The CrashLoopBackOff status specifically indicates the kubelet has entered this backoff pattern for a container repeatedly exiting shortly after starting; diagnosing it requires inspecting the container's exit code and logs, since the backoff mechanism itself only reflects that restarts are occurring, not why the underlying container keeps failing.

kubectl logs web-abc123 --previous
kubectl describe pod web-abc123

Interpreting Exit Codes and Termination Signals

Common Exit Code Meanings

kubectl get pod web-abc123 -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

An exit code of 0 indicates the process exited successfully; a non-zero application-defined code indicates an application-level failure; 137 (128 + SIGKILL's signal number 9) frequently indicates the container was killed, either by the kubelet issuing SIGKILL after the grace period expired, or by the Linux OOM killer terminating the process for exceeding its memory limit.

kubectl get pod web-abc123 -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# OOMKilled

Restarting a Container vs. Recreating a Pod

What Restart Preserves and What It Does Not

A container restart, triggered by restartPolicy or a failed liveness probe, replaces only the failed container's process within the same pod, preserving the pod's IP address, its assigned node, and any emptyDir volume contents, since the pod object itself is not deleted, only the container process within it.

volumes:
  - name: cache
    emptyDir: {}

When Full Pod Recreation Is Required

A pod is recreated entirely, rather than merely restarted, when the pod itself is deleted (by the ReplicaSet controller replacing it, by an eviction, or by a manual kubectl delete pod), which does assign a new IP address, potentially a different node, and starts with fresh emptyDir contents, representing a categorically different and more disruptive recovery event than an in-place container restart.

Init Containers and Restart Behavior

Sequential Execution Before the Main Container

initContainers:
  - name: wait-for-db
    image: busybox
    command: ["sh", "-c", "until nc -z db 5432; do sleep 2; done"]

Init containers run sequentially to completion before any main container starts, and a failing init container causes the entire pod to be retried from the beginning of the init sequence according to the pod's restartPolicy, meaning a persistently failing init container (an unreachable dependency check) produces the same CrashLoopBackOff-style pattern applied to pod startup as a whole, rather than to the main container alone.

Debugging Without Triggering Recovery

Ephemeral Containers

kubectl debug -it web-abc123 --image=busybox --target=web

Ephemeral containers are attached to an already-running pod specifically for interactive debugging, without restarting the pod or its existing containers, letting an operator inspect a live, potentially still-failing pod's filesystem and network namespace without disturbing the very failure state being investigated.

Relationship to Replica Reliability and the Reliability Model

Pod recovery basics form the innermost, fastest-acting recovery layer beneath the coarser-grained pod replacement handled by replica reliability: a container restart within an existing pod is attempted first, and only when that mechanism cannot resolve the problem, or the pod itself is removed entirely, does the ReplicaSet-level reconciliation described elsewhere take over, both layers together implementing the broader reliability model's principle of minimizing mean time to recovery through automated, tiered correction.