Kubernetes Scheduling Constraint Practice

Kubernetes Scheduling Constraint Practice is the set of authoring and operational conventions that keep placement rules — node affinity, pod affinity/anti-affinity, topology spread constraints, taints and tolerations — maintainable, predictable, and free of the subtle failure modes that arise as constraints accumulate and interact across a growing manifest base. Individually, each constraint mechanism is straightforward to understand; the practical difficulty in real clusters comes from combining several of them across many workloads over time, where a change made in isolation for one workload can silently interact with constraints already in place for another.

Good constraint practice treats placement rules as a coherent system requiring the same discipline as any other piece of critical configuration — deliberate design, testing before rollout, and periodic review — rather than a collection of independently authored, rarely revisited YAML fragments.

Preferring Soft Constraints by Default

Hard Constraints as a Deliberate, Reviewed Choice

Required (hard) constraints — requiredDuringSchedulingIgnoredDuringExecution affinity, DoNotSchedule topology spread, unmatched taints — can leave Pods permanently unschedulable if the assumptions behind them stop holding (a labeled node pool is removed, a zone becomes temporarily unavailable). Defaulting to preferred (soft) variants unless a genuine hard requirement exists keeps workloads resilient to environmental changes that were not anticipated when the constraint was originally written.

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: disktype
              operator: In
              values: ["ssd"]

Reserving Hard Constraints for Genuine Correctness Requirements

Hard constraints are appropriate specifically when violating them would produce incorrect behavior rather than merely suboptimal placement — a workload that genuinely cannot function without GPU access should hard-require it, while a workload that merely performs better with SSD storage should only prefer it.

Avoiding Constraint Sprawl

Consolidating Related Rules

Rather than layering many narrow, individually-authored constraints onto a Pod template over time (as different concerns arise from different contributors), periodically consolidating them into a smaller, well-documented set makes the resulting placement policy easier to reason about as a whole, and easier to spot unintended interactions within.

Documenting the Intent Behind Non-Obvious Constraints

A constraint whose purpose is not self-evident from its labels alone (why exactly does this workload require zone in [a, b] but not c?) benefits from an accompanying comment or annotation explaining the reasoning, since the underlying business or technical justification is easily lost once the original author moves on and the constraint remains in place indefinitely.

# Zones a and c host our primary database read replicas;
# co-locating here reduces cross-zone query latency.
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 90
        preference:
          matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: ["us-east-1a", "us-east-1c"]

Testing Constraint Changes Before Production

Staging Environment Validation

A new or modified constraint should be validated in a staging environment with a topology reasonably representative of production (similar number of zones, similar node labeling) before being applied broadly, since constraint interactions that are individually satisfiable in a small test cluster can behave differently at production scale and topology.

Dry-Run and Simulated Scheduling

kubectl apply --dry-run=server -f deployment.yaml

While a server-side dry run validates schema correctness, it does not simulate actual scheduling outcomes; confirming a constraint change produces the intended placement still requires applying it (in staging, or as a canary in production) and observing actual Pod distribution afterward.

Auditing Constraints Across a Fleet

Detecting Replica Counts Exceeding Topology Domains

A recurring, easily automated check compares each workload's replica count against the number of distinct topology domains its anti-affinity or topology spread constraints target, flagging cases where a hard constraint's implicit ceiling (one replica per domain, for kubernetes.io/hostname-scoped required anti-affinity) has been exceeded by a scaling change made without revisiting the constraint.

kubectl get deployments -A -o json | \
  jq -r '.items[] | select(.spec.replicas > (.spec.template.spec.topologySpreadConstraints // [] | length)) | .metadata.name'

Reviewing Constraint Changes in Pull Requests

Because placement constraint changes can have cluster-wide, not-immediately-visible effects, teams commonly flag changes to affinity, tolerations, and topologySpreadConstraints fields for specific reviewer attention in pull requests, distinct from more routine changes like environment variable or image tag updates.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: codartium-constraint-practice-example
spec:
  replicas: 3
  selector:
    matchLabels:
      app: codartium-api
  template:
    metadata:
      labels:
        app: codartium-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: "kubernetes.io/hostname"
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: codartium-api
      containers:
        - name: api
          image: codartium/api:latest