Kubernetes Job Failure Management

Kubernetes Job Failure Management is the collection of mechanisms a batch/v1 Job uses to detect, classify, retry, and ultimately either recover from or terminally fail on Pod-level errors. Because Jobs run finite work rather than long-lived services, failure has a different meaning than it does for a Deployment: a failed Pod is not simply restarted forever, but is weighed against explicit budgets and policies that decide whether the Job should try again or give up.

Failure management spans several layers: how an individual Pod is judged to have failed, how that failure is counted against the Job's retry budget, how long the Job is allowed to keep trying, and how the Job's terminal failure state is surfaced to operators and automation.

Detecting Pod Failure

Container Exit Codes

A container that exits with a non-zero status code is considered failed. Combined with restartPolicy, this determines whether the failure is handled in place (OnFailure, kubelet restarts the container) or causes the entire Pod to be marked Failed (Never, no in-place restart).

Infrastructure-Induced Failures

Pods can also fail for reasons unrelated to application logic: node eviction under resource pressure, node failure, preemption, or manual deletion. These produce Pod conditions such as DisruptionTarget that podFailurePolicy can specifically recognize and treat differently from an application-level crash.

Retry Budgets

backoffLimit

.spec.backoffLimit (default 6) caps the total number of Pod failures the Job will tolerate before transitioning to Failed with reason BackoffLimitExceeded. Each retry is delayed by an exponentially increasing backoff (starting at 10 seconds and doubling up to a ceiling of 6 minutes), which prevents a rapidly crash-looping workload from hammering the API server with Pod creation requests.

backoffLimitPerIndex

For Indexed Jobs, .spec.backoffLimitPerIndex gives each completion index its own independent failure budget rather than sharing one budget across the whole Job. This isolates a persistently broken shard's failures from consuming the retry allowance that healthy shards would otherwise use, and pairs with .spec.maxFailedIndexes to cap how many indices are allowed to exhaust their budget before the whole Job is failed.

activeDeadlineSeconds

Independent of failure counting, .spec.activeDeadlineSeconds bounds total wall-clock time. If exceeded, the Job is terminated and marked Failed with reason DeadlineExceeded, regardless of how much of the backoff budget remained unused.

Classifying Failures with podFailurePolicy

.spec.podFailurePolicy lets a Job apply different actions depending on why a Pod failed, evaluated against container exit codes or Pod conditions:

• FailJob: immediately fails the whole Job, bypassing further retries, typically used for exit codes that indicate a non-retryable application error (bad input, invalid configuration).
• Ignore: does not count the failure against backoffLimit at all, and still triggers a retry — used for infrastructure-induced failures like node preemption that say nothing about whether the work itself is broken.
• Count: the default behavior, counting the failure toward the backoff budget like any ordinary failure.

podFailurePolicy:
  rules:
    - action: FailJob
      onExitCodes:
        containerName: worker
        operator: In
        values: [42]
    - action: Ignore
      onPodConditions:
        - type: DisruptionTarget

This lets a Job distinguish "this input is fundamentally invalid, stop trying" from "the node disappeared, that's not the workload's fault, try again without penalty."

Terminal Failure

Conditions and Reasons

A failed Job surfaces a Failed condition in .status.conditions with a reason field explaining why: BackoffLimitExceeded, DeadlineActiveDeadlineExceeded, PodFailurePolicy, or FailedIndexes for Indexed Jobs that exceeded maxFailedIndexes.

Operator Response

kubectl get job codartium-batch-import -o jsonpath='{.status.conditions[?(@.type=="Failed")]}'
kubectl describe job codartium-batch-import
kubectl logs -l job-name=codartium-batch-import --previous --all-containers

Inspecting Pod logs with --previous is often necessary because the failing Pods themselves may already have been garbage collected or superseded by later retries by the time an operator investigates.

Retrying After a Terminal Failure

A Job that has reached Failed will not retry further on its own; a fixed Job object does not resume automatically once the retry budget is spent. Operators or automation must create a new Job (often after fixing the underlying issue) to attempt the work again.

Example

apiVersion: batch/v1
kind: Job
metadata:
  name: codartium-failure-example
spec:
  completions: 5
  parallelism: 2
  backoffLimit: 3
  activeDeadlineSeconds: 600
  podFailurePolicy:
    rules:
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget
      - action: FailJob
        onExitCodes:
          containerName: worker
          operator: In
          values: [42]
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: codartium/worker:latest