Kubernetes Job Retry Management

Kubernetes Job Retry Management is the mechanism by which a batch/v1 Job automatically re-attempts failed work by replacing failed Pods with new ones, governed by configurable limits and timing so that retries help a workload recover from transient problems without allowing a persistently broken workload to consume unbounded cluster resources. Retry management sits between failure detection and terminal failure: every retry is a bet that the next attempt will succeed, and the Job controller places a bounded number of those bets before giving up.

Retries in Kubernetes Jobs operate at the Pod level — a failed Pod is not repaired in place (unless restartPolicy: OnFailure restarts a container within the same Pod); instead, the controller creates a brand-new Pod to attempt the same unit of work again.

How Retries Are Triggered

Pod Failure as the Trigger

Whenever a Pod owned by a Job terminates in a failed state — because a container exited non-zero under restartPolicy: Never, or because the Pod was evicted or deleted before completing — the Job controller observes the failure and, provided the retry budget is not exhausted, creates a replacement Pod for the same unit of work (the same index, for Indexed Jobs, or simply one more attempt toward the aggregate completions count for NonIndexed Jobs).

restartPolicy: OnFailure vs Never

• OnFailure retries happen inside the same Pod: the kubelet restarts the failing container, and these in-place restarts are tracked by the container's own restart count, separate from the Job's backoffLimit accounting, though a Pod that never manages to succeed will eventually still be considered failed at the Pod level if the container keeps crashing indefinitely relative to the Pod's own backoff.
• Never retries happen at the Pod level: each failure produces an entirely new Pod, which is the model most directly and visibly governed by backoffLimit.

Bounding Retries

backoffLimit

The primary lever: .spec.backoffLimit (default 6) is the maximum number of Pod failures the Job will tolerate in total before it transitions to Failed. Every counted failure decrements the remaining budget; once it reaches zero, no further replacement Pods are created.

Exponential Backoff Timing

Between retries, the controller inserts a delay that grows exponentially with each successive failure — starting around 10 seconds and doubling with each retry up to a capped maximum (6 minutes). This prevents a workload stuck in a fast failure loop from generating an unbounded rate of Pod creation calls against the API server, and gives transient conditions (a dependency restarting, a brief network partition) time to resolve between attempts.

Per-Index Retry Budgets

For completionMode: Indexed Jobs, .spec.backoffLimitPerIndex allocates a separate retry budget to each completion index rather than pooling all retries into one shared counter. This is important for large fan-out Jobs: without per-index budgets, one persistently broken shard could exhaust the entire Job's retry allowance and stop other, healthy shards from getting retried at all. .spec.maxFailedIndexes complements this by capping how many indices are allowed to exhaust their individual budget before the whole Job is failed, rather than requiring every single index to succeed.

Refining What Counts as Retryable

.spec.podFailurePolicy changes retry behavior beyond simple counting:

• A rule with action Ignore causes a matching failure to trigger a retry without consuming any of the backoffLimit budget — appropriate for infrastructure-driven failures like node preemption, where the failure says nothing about whether the workload itself is broken.
• A rule with action FailJob skips retries altogether for a matching failure, useful for exit codes that indicate the work is fundamentally unrecoverable (malformed input, a configuration error) where further attempts would only waste time and resources.

podFailurePolicy:
  rules:
    - action: Ignore
      onPodConditions:
        - type: DisruptionTarget
    - action: FailJob
      onExitCodes:
        containerName: worker
        operator: In
        values: [78]

Observing Retry Activity

kubectl get pods -l job-name=codartium-batch-import
kubectl get job codartium-batch-import -o jsonpath='{.status.failed}'
kubectl describe job codartium-batch-import

kubectl describe job surfaces recent events, including Pod creation and failure events, which is typically the fastest way to see the retry history of a Job without manually cross-referencing individual Pod objects.

Example

apiVersion: batch/v1
kind: Job
metadata:
  name: codartium-retry-example
spec:
  completions: 10
  parallelism: 3
  completionMode: Indexed
  backoffLimitPerIndex: 3
  maxFailedIndexes: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: codartium/worker:latest