Kubernetes Reliability and Availability Basics

Kubernetes Reliability and Availability Basics covers the configuration primitives and operational practices that allow applications running on Kubernetes to tolerate failures, individual Pod crashes, node outages, planned maintenance, without an interruption in service visible to their users. Reliability on Kubernetes is not automatic simply by virtue of running on the platform; it results from deliberately configuring replication, disruption tolerance, health signaling, and control plane redundancy so that the self-healing behavior the platform provides is actually exercised correctly when failures occur.

Replication as the Foundation of Availability

Running Multiple Replicas

The most fundamental reliability practice is running more than one replica of a workload through a Deployment or StatefulSet, so that the failure of any single Pod, or the node hosting it, leaves remaining replicas available to continue serving traffic while a replacement is created.

where p is the probability of an individual replica being unavailable and n is the replica count, illustrating why even a modest increase in replica count meaningfully improves availability when failures are largely independent.

Spreading Replicas Across Failure Domains

Replication alone is insufficient if all replicas can fail together; anti-affinity rules and topology spread constraints are used to distribute replicas across distinct nodes and, where the cluster spans them, distinct availability zones, so that a single node or zone failure does not remove every replica simultaneously.

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: codartium-api

Health Signaling

Probes as the Input to Self-Healing

Liveness and readiness probes are the mechanism through which a workload communicates its own health to the platform; without accurate probes, the kubelet has no reliable signal to detect a hung or broken container, and Services have no reliable signal to withhold traffic from an instance that is not yet, or no longer, able to serve requests correctly.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

Graceful Shutdown

Correctly handling SIGTERM and completing in-flight work before exiting, combined with prompt removal from Service endpoints on termination, prevents rolling updates and node maintenance from causing dropped requests, even though individual Pods are being replaced.

PodDisruptionBudget

Bounding Voluntary Disruption

A PodDisruptionBudget (PDB) constrains how many replicas of a workload may be voluntarily disrupted at once, during node drains, cluster upgrades, or cluster-autoscaler-initiated evictions, ensuring that operations under the cluster operator's control cannot reduce availability below an acceptable threshold, even while involuntary disruptions (node crashes) remain outside its scope.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: codartium-api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: codartium-api

Interaction with Cluster Operations

Any component performing a voluntary eviction, such as the Cluster Autoscaler scaling down a node or an administrator draining a node for maintenance, must respect active PodDisruptionBudgets, deferring or skipping evictions that would violate the configured budget until a safe opportunity arises.

kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data
kubectl get pdb -n codartium-team

Control Plane and etcd Availability

High-Availability Control Plane

Because the control plane's own availability directly bounds the cluster's ability to schedule, heal, and reconcile workloads, production clusters typically run multiple replicas of the API server, controller manager, and scheduler, with the API server fronted by a load balancer and the controller manager and scheduler coordinating through leader election.

etcd Quorum

etcd availability requires maintaining quorum among its member nodes; deploying an odd number of etcd members, most commonly three or five, and distributing them across separate failure domains ensures the cluster's source of truth remains available and consistent even if a minority of members are lost.

Rollout Safety

Controlled Rollouts

Conservative rolling update parameters, maxUnavailable set low or to zero for critical services, combined with accurate readiness probes gating the pace of the rollout, prevent a bad deployment from taking down capacity faster than its problems can be detected.

Fast Rollback

Retaining sufficient revision history and validating rollout health through kubectl rollout status or automated checks allows a faulty release to be identified and rolled back quickly, minimizing the duration of any availability impact from a bad change.

kubectl rollout status deployment/codartium-api --timeout=90s
kubectl rollout undo deployment/codartium-api

Combining Practices into a Reliability Posture

No single mechanism, replication, probes, PodDisruptionBudgets, or control plane redundancy, provides availability on its own; reliability on Kubernetes emerges from these mechanisms working together, so that failures at every layer, individual containers, nodes, availability zones, and voluntary maintenance operations, are each addressed by a corresponding, deliberately configured safeguard.