Kubernetes Job Success Management

Kubernetes Job Success Management is the set of rules and mechanisms that determine when an individual Pod created by a batch/v1 Job is counted as successful, and how those individual successes accumulate into overall Job success. While failure handling in Kubernetes Jobs receives significant attention because it drives retries and backoff, success determination is equally deliberate: Kubernetes must decide, for each Pod, whether its termination represents completed work, and it must do so consistently across container restarts, multi-container Pods, and Indexed Job semantics.

A Pod's success is fundamentally derived from its containers' exit codes and restart policy, but the way that Pod-level success rolls up into Job-level success differs depending on completion mode, and can be further refined using Pod failure policies that reclassify certain outcomes.

Pod-Level Success

Container Exit Codes

A container is considered to have terminated successfully when its process exits with status code 0. For a Pod to be counted as Succeeded, every container in the Pod (excluding sidecars implemented as restartable init containers, which have their own lifecycle rules) must terminate successfully; the Pod does not need to be actively running, since Job Pods are expected to run to completion rather than indefinitely.

restartPolicy Interaction

• With restartPolicy: Never, any container failure terminates the Pod in a failed state immediately; there is no in-place retry, so success requires every container to exit cleanly on the first attempt within that Pod.
• With restartPolicy: OnFailure, a failing container is restarted in place by the kubelet, and the Pod is only marked Succeeded once all containers have exited with code 0 on some attempt — intermediate failed attempts within the same Pod do not, by themselves, fail the Pod, though they do count against the container's own restart count.

Multi-Container Pods

When a Job's Pod template defines multiple containers, all of them must succeed for the Pod to be counted as successful. This matters for sidecar patterns (log shippers, proxies) running alongside a primary batch container — if the sidecar does not exit cleanly when the main container finishes, the Pod may never be marked successful, which is why sidecars in Job Pods are typically implemented using native sidecar support (restartable init containers) so their lifecycle is tied correctly to the main container's completion.

Job-Level Success Rollup

NonIndexed Jobs

Successful Pods are simply counted in .status.succeeded. The Job is complete once this counter reaches .spec.completions; which specific Pod produced which success is not tracked individually.

Indexed Jobs

Success is tracked per completion index, recorded in .status.completedIndexes. An index is considered done as soon as any Pod for that index succeeds, and the controller will not create further Pods for an index that already has a successful completion, even if parallelism allows for more capacity — this avoids redundant work for shards that already finished.

Work Queue Jobs

Since completions is unset, Job-level success is inferred rather than counted: the Job is complete once at least one Pod has exited successfully and no Pods remain active, on the assumption that the last surviving worker observed the queue was empty before exiting cleanly.

Refining Success Determination with podFailurePolicy

.spec.podFailurePolicy can reclassify what would otherwise look like a failure into a non-counting, ignorable event — for example, a Pod evicted due to node maintenance is not really evidence that the work failed, so a rule with action Ignore on DisruptionTarget conditions lets the controller simply retry without treating the eviction as a strike against backoffLimit. This keeps the Job's failure accounting focused on genuine application-level failures rather than infrastructure noise, which indirectly makes eventual success easier to reach without inflating the retry budget on transient causes.

podFailurePolicy:
  rules:
    - action: Ignore
      onPodConditions:
        - type: DisruptionTarget
    - action: Count
      onExitCodes:
        containerName: worker
        operator: NotIn
        values: [0]

Observing Success

kubectl get job codartium-batch-import -o jsonpath='{.status.succeeded}'
kubectl get job codartium-indexed-shards -o jsonpath='{.status.completedIndexes}'
kubectl get pods -l job-name=codartium-batch-import --field-selector=status.phase=Succeeded

Example

apiVersion: batch/v1
kind: Job
metadata:
  name: codartium-success-example
spec:
  completions: 4
  parallelism: 2
  completionMode: Indexed
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: worker
          image: codartium/worker:latest
          command: ["sh", "-c", "run-task && exit 0"]