Kubernetes Batch Reliability Basics

Kubernetes Batch Reliability Basics is the set of reliability considerations specific to Job and CronJob workloads, where success is defined by completion rather than continuous availability, covering retry and backoff limits, fine-grained failure policies, deadline enforcement, and the concurrency control that governs scheduled, recurring batch execution.

Retry and Backoff for Job Pods

backoffLimit and Exponential Retry Delay

apiVersion: batch/v1
kind: Job
metadata:
  name: data-import
spec:
  backoffLimit: 4
  template:
    spec:
      restartPolicy: OnFailure

A Job's backoffLimit bounds how many times a failed pod is retried before the Job itself is marked failed, with retries spaced by the same exponential backoff algorithm the kubelet applies to container restarts, preventing a persistently failing batch task from retrying indefinitely while still tolerating a reasonable number of transient failures.

restartPolicy: OnFailure vs. Never

template:
  spec:
    restartPolicy: Never

OnFailure restarts the same pod in place on failure, counted against backoffLimit; Never instead creates an entirely new pod for each retry attempt, which is useful when a failed pod's logs or final state should be preserved for inspection rather than overwritten by an in-place restart.

Completions, Parallelism, and Indexed Jobs

Coordinating Multiple Successful Completions

spec:
  completions: 10
  parallelism: 3

For batch work requiring multiple successful pod completions, completions sets the total required, and parallelism bounds how many run concurrently, with the Job controller reliably tracking exactly how many have succeeded so far and creating new pods only as needed to reach the target, tolerating individual pod failures along the way without losing count of overall progress.

Indexed Completion Mode

spec:
  completionMode: Indexed
  completions: 10

Indexed mode assigns each pod a unique, stable completion index (available as an environment variable), letting a batch task partition work deterministically across pods and, critically, letting a failed indexed pod be retried without disturbing or duplicating the work already completed by other indices, a meaningfully more reliable pattern than relying on parallel workers to self-coordinate partitioning.

Fine-Grained Failure Handling

Pod Failure Policy

spec:
  podFailurePolicy:
    rules:
      - action: FailJob
        onExitCodes:
          containerName: main
          operator: In
          values: [42]
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget

podFailurePolicy distinguishes failure causes that should immediately fail the entire Job (an application-defined "unrecoverable error" exit code) from failures that should be ignored and not counted against backoffLimit at all (a pod evicted due to node disruption, which is not the batch task's own fault), giving considerably more precise control than treating every failure identically.

Deadline Enforcement

activeDeadlineSeconds

spec:
  activeDeadlineSeconds: 3600

activeDeadlineSeconds bounds the total wall-clock time a Job may remain active before being terminated and marked failed regardless of backoffLimit or completion progress, protecting against a batch task that is technically still retrying but has clearly exceeded any reasonable expected duration, distinct from and complementary to the retry-count-based backoffLimit.

CronJob Concurrency and Schedule Reliability

concurrencyPolicy

apiVersion: batch/v1
kind: CronJob
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid

Allow (the default) permits overlapping runs if a prior scheduled Job has not finished; Forbid skips a new scheduled run entirely if the previous one is still active; Replace cancels the still-running prior Job and starts the new one; the correct choice depends on whether the batch task is safe to run concurrently with itself, a critical reliability decision for any task with side effects that are not safe to duplicate.

startingDeadlineSeconds for Missed Schedules

spec:
  startingDeadlineSeconds: 300

If the CronJob controller itself is unavailable or delayed (a control plane outage) past a scheduled trigger time, startingDeadlineSeconds bounds how late a missed run may still be started before it is abandoned entirely, preventing a large backlog of stale, no-longer-relevant scheduled runs from all firing simultaneously once the controller recovers.

Idempotency as the Underlying Requirement

Why Retries Demand Idempotent Task Design

Every retry mechanism described above assumes the batch task itself is safe to run more than once for the same logical unit of work, an assumption that must be satisfied by the task's own implementation (using upsert semantics, idempotency keys, or exactly-once side-effect design) rather than by Kubernetes, since Kubernetes provides the retry mechanics but has no way to verify that a retried task will not produce incorrect duplicated side effects.

Relationship to Deployment and Daemon Reliability Basics

Batch reliability basics address a fundamentally different success criterion than the continuous-availability focus of deployment and daemon reliability basics, completion of a bounded unit of work rather than sustained readiness, and its mechanisms, backoff limits, indexed completions, pod failure policies, deadlines, concurrency control, reflect the broader reliability model's principle of tailoring recovery mechanics to the specific failure and success semantics each distinct workload topology actually requires.