Kubernetes Job Failure Control

Kubernetes Job Failure Control is the set of rules determining when a Job as a whole, not merely an individual Pod, is declared permanently failed, and what happens once that terminal determination is made. Where completion control governs how individual Pod outcomes are counted, failure control governs the aggregate decision and its consequences: a Job that has failed does not retry itself, does not create further Pods, and requires explicit external action to run again.

backoffLimit as the Primary Failure Trigger

Aggregate Retry Exhaustion

For non-indexed Jobs, spec.backoffLimit counts failed Pod attempts across the whole Job; once this count is exceeded, the controller marks the Job Failed with reason BackoffLimitExceeded and stops creating any further Pods, regardless of how many completions might still theoretically be achievable.

apiVersion: batch/v1
kind: Job
metadata:
  name: failure-control-example
spec:
  backoffLimit: 6
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: registry.example.com/worker:1.0.0

Per-Index Failure Limits for Indexed Jobs

backoffLimitPerIndex

For Indexed completion mode, spec.backoffLimitPerIndex allows each index to exhaust its own retry budget independently rather than sharing a single Job-wide counter, so a persistently failing index does not consume retry attempts that other, healthy indexes would otherwise use.

spec:
  completionMode: Indexed
  completions: 10
  parallelism: 10
  backoffLimitPerIndex: 2

maxFailedIndexes

spec.maxFailedIndexes bounds how many distinct indexes are allowed to exhaust their per-index backoff before the entire Job is marked Failed, letting a bounded number of individually unrecoverable partitions be tolerated without failing the whole batch.

spec:
  maxFailedIndexes: 2

activeDeadlineSeconds as an Independent Failure Path

Wall-Clock Limit Regardless of Retry Budget

spec.activeDeadlineSeconds fails a Job with reason DeadlineExceeded purely based on elapsed time since the Job became active, independent of backoffLimit. A Job can be terminated this way even if it has retries remaining, making it the appropriate control for bounding total run time rather than total attempt count.

spec:
  activeDeadlineSeconds: 1800

No Self-Retry After Terminal Failure

The Job Does Not Restart Itself

Once a Job reaches the Failed condition, the controller takes no further corrective action; it does not create new Pods, and the Job object remains in its failed state until an operator deletes and recreates it, or, in the case of a CronJob-managed Job, until the next scheduled invocation creates an entirely new Job object.

kubectl get job failure-control-example -o jsonpath='{.status.conditions[?(@.type=="Failed")].status}'

Downstream Effects on CronJob

Failure Visibility to the Owning Controller

When a Job created by a CronJob fails, the CronJob controller records this in its own status but does not itself retry the failed run; the next Job is only created at the next scheduled time, meaning a failed run's remediation is either manual or deferred entirely to the following scheduled occurrence.

kubectl get cronjob -o jsonpath='{.items[0].status.lastScheduleTime}'

Job Failure Control Diagram

Understanding that these two paths, retry exhaustion and deadline expiration, are independent and that neither triggers any automatic remediation is essential for designing monitoring and alerting around batch workloads, since a failed Job is a terminal state requiring active intervention, not a transient condition the system will resolve on its own.