10.19 Kubernetes Job Failure Control

Kubernetes Job Failure Control is the set of configuration options and tracking mechanisms a Job uses to determine how individual Pod failures are handled, how many retries are permitted before giving up, and under what precise conditions the Job as a whole should be marked as permanently failed rather than continuing to attempt completion.

Pod-Level Failure and Retry

Restart Policy Requirements

Job-managed Pods must use a restartPolicy of either OnFailure or Never, since Always would conflict with the Job's fundamental run-to-completion model; under OnFailure, a failing container is restarted in place within the same Pod, while under Never, a failed Pod is left terminated and an entirely new Pod is created to retry the work instead.

backoffLimit as the Retry Ceiling

The backoffLimit field specifies the maximum number of retries the Job controller will attempt before giving up and marking the Job as failed, with each retry subject to an exponential backoff delay, ensuring a persistently failing task does not consume cluster resources through unbounded rapid retry attempts.

Pod Failure Policy

Fine-Grained Failure Classification

The podFailurePolicy field allows a Job to define explicit rules distinguishing different categories of Pod failure, matching against specific container exit codes or Pod conditions, and specifying whether a given failure category should count toward the backoffLimit, be ignored entirely, or immediately fail the Job regardless of remaining retry budget.

Handling Disruption-Caused Failures Separately

A common use of pod failure policy is to prevent failures caused by voluntary disruptions, such as a Pod being evicted during node maintenance, from counting against the Job's retry budget in the same way as a genuine application-level failure would, since a disruption-caused failure reflects infrastructure conditions rather than a problem with the task itself.

activeDeadlineSeconds as a Time-Based Failure Trigger

Enforcing an Overall Execution Time Limit

The activeDeadlineSeconds field establishes a maximum duration the Job is permitted to remain active, measured from when it starts; if this deadline is exceeded regardless of retry count or completion progress, the Job is terminated and marked as failed, providing protection against tasks that hang indefinitely without ever explicitly failing on their own.

Interaction With Ongoing Retries

Because this deadline applies to the Job's total elapsed active time rather than to any individual Pod attempt, a Job that has been retrying within its backoffLimit can still be forcibly failed by exceeding activeDeadlineSeconds, meaning both limits operate independently and either one reaching its threshold results in the Job's failure.

Failed Condition and Terminal State

Marking the Job as Failed

Once a Job's failure criteria are met, whether through exhausting backoffLimit, exceeding activeDeadlineSeconds, or triggering an immediate-fail rule under podFailurePolicy, its status reflects a Failed condition set to true, representing a permanent terminal state from which the Job will not automatically recover or retry further.

No Automatic Recovery From Failure

Once marked failed, a Job does not attempt any further Pod creation, and reviving the underlying task requires either manual intervention, such as creating a new Job to reattempt the work, or reliance on a higher-level controller such as a CronJob that will create a fresh Job instance on its next scheduled invocation.

Visibility Into Failure Causes

Pod Status and Events as Diagnostic Sources

Diagnosing why a Job has failed typically requires examining the status and termination reason of its individual failed Pods, alongside any Kubernetes events recorded against the Job or its Pods, since the Job's own top-level status reports only the aggregate failure outcome rather than a detailed explanation of the underlying cause.