Kubernetes Batch Workloads

Kubernetes Batch Workloads are workloads that run to completion rather than continuously, processing a defined unit of work and terminating once that work is finished, in contrast to long-running services such as web servers that are expected to run indefinitely. Kubernetes supports batch processing through the Job and CronJob resources, which track Pods to successful completion, handle retries on failure, and, in the case of CronJob, initiate that work on a recurring schedule.

Job

Purpose and Semantics

A Job creates one or more Pods and ensures a specified number of them terminate successfully. Unlike a Deployment, which treats a Pod exiting as a failure to be corrected by restarting it into a long-running state, a Job treats successful termination as the intended outcome and stops creating replacement Pods once its completion target is met.

apiVersion: batch/v1
kind: Job
metadata:
  name: codartium-migration
spec:
  completions: 1
  backoffLimit: 4
  template:
    spec:
      containers:
        - name: migrate
          image: codartium/migrator:2.0.0
          command: ["python", "migrate.py"]
      restartPolicy: Never

Completion Modes

• Non-parallel Jobs: A single Pod is created, and the Job is complete once that Pod succeeds.
• Parallel Jobs with a fixed completion count: The completions field specifies how many Pods must succeed in total, while parallelism bounds how many run concurrently.
• Parallel Jobs with a work queue: Multiple Pods coordinate independently, commonly against an external queue, and the Job is considered complete once any one Pod succeeds and all Pods have terminated.

spec:
  completions: 10
  parallelism: 3

Retry and Backoff

If a Pod created by a Job fails, the Job controller creates a replacement Pod, up to the number of attempts allowed by backoffLimit, with an exponentially increasing delay between attempts. Once this limit is exceeded, the Job is marked as failed and no further Pods are created.

Indexed Jobs

An indexed completion mode assigns each Pod in a parallel Job a fixed completion index, exposed to the container as an environment variable, allowing each Pod to determine which portion of a larger, partitionable task it is responsible for without external coordination.

spec:
  completionMode: Indexed
  completions: 5
  parallelism: 5

Active Deadlines and TTL Cleanup

A Job can specify activeDeadlineSeconds to bound its total runtime, after which it is terminated regardless of completion state, and ttlSecondsAfterFinished to have the Job and its Pods automatically garbage collected a defined period after completion, avoiding manual cleanup of finished batch resources.

CronJob

Purpose

A CronJob creates Jobs on a repeating schedule, expressed using standard cron syntax, and is used for periodic batch tasks such as generating reports, running backups, or performing scheduled maintenance.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: codartium-nightly-report
spec:
  schedule: "30 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: report
              image: codartium/report:1.4.0
          restartPolicy: OnFailure

Concurrency Policy

The concurrencyPolicy field determines how overlapping runs are handled: Allow permits concurrent Job instances, Forbid skips a new run if the previous one has not finished, and Replace cancels the still-running Job in favor of the new scheduled instance.

Schedule Semantics and Missed Runs

A CronJob tracks a startingDeadlineSeconds window within which a missed scheduled run, due to control plane downtime or other delay, may still be started; beyond this window, the missed execution is skipped rather than run late, avoiding a backlog of delayed executions all firing at once.

Batch Workloads and Node Scheduling

Because batch workloads are often bursty, running a large number of Pods briefly and then none at all, clusters running significant batch workloads frequently rely on cluster autoscaling to provision additional nodes for the duration of a burst and scale back down once the batch work completes, and use resource requests carefully to allow the scheduler to pack batch Pods efficiently alongside longer-running services.

kubectl get jobs
kubectl logs job/codartium-migration
kubectl delete job codartium-migration --cascade=foreground

Comparison with Long-Running Controllers

Where Deployment, StatefulSet, and DaemonSet all treat Pod termination as an event to correct, Job and CronJob treat termination as the expected, successful outcome of the work being done, and this difference in what "success" means is what fundamentally distinguishes batch workload management from service workload management within the same platform.