Kubernetes Batch Workload Guidelines

Kubernetes Batch Workload Guidelines describe the practices for running finite, run-to-completion work on Kubernetes — one-off tasks, scheduled jobs, and parallel processing workloads — using the Job and CronJob resources rather than the continuously-running abstractions (Deployment, StatefulSet) designed for long-lived services. Batch workloads have fundamentally different lifecycle semantics: success is defined by completion, not by continuous readiness, and failure handling must account for retries, backoff, and cleanup in ways that a service workload does not need to consider.

The Job Resource

Completion Semantics

A Job creates one or more Pods and tracks them until a specified number reach successful completion (completions), at which point the Job itself is marked complete and no further Pods are created. This is a fundamentally different success condition from a Deployment, which has no concept of "done" — a Deployment's desired state is perpetual, a Job's desired state is a finite amount of completed work.

Parallelism

The parallelism field controls how many Pods may run concurrently while working toward the required completions. Setting parallelism below completions processes work in waves; setting them equal starts all work simultaneously, appropriate when the work items are independent and the cluster has capacity to absorb the full burst.

Restart Policy

Job Pods must use restartPolicy: Never or restartPolicy: OnFailure, never Always — a batch workload's whole purpose is to terminate, so the restart policies designed for continuously-running Pods are inapplicable and will be rejected by the API for a Job's Pod template.

Retry and Failure Handling

backoffLimit

backoffLimit caps the number of retries before the Job is marked as failed outright, with retry delays following exponential backoff. Without a bound here, a Job whose underlying task is permanently broken (a bad image, a code bug) would retry indefinitely, consuming cluster resources without ever making progress.

Pod Failure Policy

podFailurePolicy allows distinguishing between failure types — for example, treating an OOM-killed Pod as a signal to retry, while treating a specific non-zero exit code as a signal to fail the Job immediately without further retries. This prevents wasting retry budget on failures that are deterministic and will never succeed on a subsequent attempt.

activeDeadlineSeconds

activeDeadlineSeconds bounds the total wall-clock time a Job is allowed to run before being forcibly terminated and marked failed, protecting against a Job that is technically still making progress but taking far longer than expected due to a stuck dependency or a performance regression.

Scheduled Execution With CronJob

Schedule Expression

A CronJob creates Job objects on a cron-formatted schedule, inheriting all of a Job's completion and retry semantics for each individual run. The schedule is evaluated in the timezone specified by the timeZone field (defaulting to the kube-controller-manager's timezone if unset), which should always be set explicitly to avoid ambiguity across cluster migrations.

Concurrency Policy

concurrencyPolicy determines what happens if a scheduled run's start time arrives while a previous run is still active: Allow runs them concurrently, Forbid skips the new run entirely, and Replace cancels the still-running Job in favor of the new one. Workloads that are not safe to run concurrently (anything performing exclusive writes to shared state) must use Forbid, not the default Allow.

Missed Schedule Handling

startingDeadlineSeconds bounds how late a missed scheduled run is still allowed to start, preventing a cluster outage or controller downtime from causing a flood of backlogged CronJob runs firing simultaneously once the controller recovers.

History Limits

successfulJobsHistoryLimit and failedJobsHistoryLimit bound how many completed Job objects are retained for inspection. Retaining a small number of failed jobs is valuable for debugging; retaining unlimited history accumulates cluster object clutter over time.

Resource and Scheduling Considerations

Resource Requests for Bursty Load

Batch workloads often run in bursts that briefly demand significant capacity. Requests should reflect actual per-Pod needs, and cluster capacity planning (or cluster autoscaler configuration) should account for the burst pattern of scheduled batch work rather than only steady-state service load.

Priority for Preemptible Work

Batch Jobs are frequently good candidates for a lower PriorityClass than user-facing services, since a batch job can generally tolerate being preempted and retried later, whereas a preempted service Pod causes an immediate customer-facing impact.

Example Configuration

apiVersion: batch/v1
kind: CronJob
metadata:
  name: codartium-nightly-report
spec:
  schedule: "0 2 * * *"
  timeZone: "UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: report-generator
              image: registry.example.com/codartium-report@sha256:ff44aa...
              resources:
                requests:
                  cpu: "1"
                  memory: "1Gi"

Practical Consequences

Well-designed batch workload configuration produces jobs that retry transient failures without wasting resources on deterministic ones, that never overlap when overlap would corrupt shared state, and that leave a clean, bounded audit trail of recent runs. Neglecting these guidelines commonly results in Jobs that retry a permanently broken task indefinitely, CronJobs that silently stack up duplicate concurrent runs against a resource that assumed exclusive access, or a flood of backlogged executions after a control plane outage overwhelming the very system the Job was meant to process on a controlled schedule.