Kubernetes Job Execution Management

Kubernetes Job Execution Management is the set of controller behaviors, state transitions, and operational controls that govern how a batch/v1 Job actually runs once it has been submitted to the API server — as distinct from how the Job is specified. Where Job Spec Structure describes the fields an author writes, Execution Management describes what the Job controller does with those fields at runtime: how it decides when to create Pods, when to retry, when to stop, and how it reports progress back through status fields and conditions.

The Job controller reconciles the running state of a Job against its desired state continuously. It watches for Pod creation, completion, and failure events, and drives the Job toward one of its terminal states — Complete, Failed, or, in newer Kubernetes versions, FailureTarget transitioning to Failed — based on the counts and policies declared in the spec.

Execution Lifecycle

Pod Creation and Scheduling

When a Job is created, the controller begins creating Pods up to the configured parallelism limit. Each Pod is scheduled independently by the default Kubernetes scheduler, meaning Pods from the same Job may land on different nodes based on resource availability, affinity rules, and taints/tolerations. The controller does not create all Pods for a Job at once when completions exceeds parallelism; instead, it maintains a steady-state number of active Pods, creating new ones as earlier Pods complete or fail, until the completion target is met.

Tracking Successes and Failures

As Pods terminate, the controller updates .status.succeeded and .status.failed counters. For NonIndexed Jobs, only the total counts matter. For Indexed Jobs, the controller additionally tracks which specific indices have succeeded, allowing it to avoid re-running indices that already completed even while other indices are still retrying.

Retry Behavior

A failed Pod does not, by itself, fail the Job. The controller creates a replacement Pod, subject to backoffLimit (or backoffLimitPerIndex for Indexed Jobs). Retries are spaced using an exponential backoff, capped at a maximum delay, to avoid a crash-looping workload from generating an unbounded burst of Pod creation requests against the API server.

Pod Failure Policy Evaluation

When podFailurePolicy is set, the controller inspects each failed Pod's container exit codes and Pod conditions against the configured rules before deciding whether the failure counts toward the backoff limit at all. A rule with action Ignore allows the controller to treat certain failures (for example, a Pod evicted due to node pressure) as neutral events that trigger a retry without consuming the failure budget, while a rule with action FailJob can short-circuit retries entirely and fail the Job immediately on a specific exit code.

Deadline Enforcement

If activeDeadlineSeconds is set, the controller tracks elapsed time since the Job became active. On exceeding the deadline, it terminates all running Pods and marks the Job Failed with reason DeadlineExceeded, regardless of how many Pods had already succeeded.

Completion and Terminal State

Once the required number of successful completions is reached (or, for work-queue-style Jobs, once all Pods have exited and at least one succeeded), the controller sets the Complete condition and stops creating new Pods. If the backoff limit or active deadline is exceeded first, it instead sets the Failed condition.

Operational Controls During Execution

Suspending and Resuming

Toggling .spec.suspend to true on a running Job causes the controller to delete all active Pods without marking the Job as failed; toggling it back to false resumes Pod creation from where tracked progress left off. This is the mechanism used by external queueing systems to hold Jobs pending resource availability without losing their identity or accumulated status.

Manual Intervention

Operators can delete individual Pods belonging to a running Job to force a retry, or delete the Job itself (with or without --cascade=orphan) to stop execution. Deleting a Job by default cascades to delete its Pods; using --cascade=orphan leaves the Pods running while removing the Job's ownership, which is occasionally used to detach a long-running Pod for manual inspection.

Observing Execution

kubectl get jobs -w
kubectl describe job codartium-batch-import
kubectl get pods -l job-name=codartium-batch-import -o wide
kubectl logs -f -l job-name=codartium-batch-import --all-containers

The -w (watch) flag on kubectl get jobs is particularly useful during execution management, since it streams status changes (succeeded and failed counters) as the controller reconciles the Job in real time.

Example: Forcing a Retry and Observing Backoff

kubectl delete pod -l job-name=codartium-batch-import,batch.kubernetes.io/job-completion-index=2
kubectl get pods -l job-name=codartium-batch-import
kubectl get job codartium-batch-import -o jsonpath='{.status}'

apiVersion: batch/v1
kind: Job
metadata:
  name: codartium-execution-example
spec:
  parallelism: 2
  completions: 4
  backoffLimit: 6
  activeDeadlineSeconds: 1200
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: codartium/worker:latest