Kubernetes Reliability Guidelines

Kubernetes Reliability Guidelines describe the practices for designing workloads and cluster configuration so that the system continues serving its function correctly through failures — node loss, zone outages, dependency degradation, and the platform's own routine maintenance — rather than treating failure as an exceptional event that only needs to be handled after it has already caused an outage. Kubernetes provides substantial self-healing machinery, but that machinery only produces reliability when workloads are designed to cooperate with it.

Redundancy as the Foundation

Multiple Replicas Are Non-Negotiable

Any workload with a user-facing availability requirement must run more than one replica; a single-replica workload has no tolerance for the Pod restart, node drain, or rescheduling event that Kubernetes' normal operation guarantees will eventually happen. Replica count should be chosen based on the number of simultaneous failures the workload needs to tolerate while still meeting its capacity needs, not simply "more than one."

Spreading Across Failure Domains

Pod anti-affinity or topology spread constraints (covered under scheduling guidelines) ensure that redundant replicas are actually spread across nodes and availability zones, since replicas concentrated on a single node or zone provide no real protection against the failure of that node or zone — redundancy in name only.

Redundancy Extends to Dependencies

A reliable frontend service backed by a single-replica database, or dependent on a single-instance external API with no fallback, inherits that dependency's reliability ceiling regardless of how well the frontend itself is architected — reliability analysis must trace through the full dependency chain, not stop at the boundary of the workload being directly modified.

Designing for Graceful Degradation

Timeouts and Circuit Breakers

Every call to a downstream dependency should have an explicit timeout, and calls to dependencies prone to intermittent failure should be protected by a circuit breaker that stops sending requests to a dependency that's clearly failing, rather than piling up requests against it and exhausting the calling service's own resources (connection pools, threads) waiting on a dependency that isn't going to respond.

Retries With Backoff and Jitter

Retrying a failed call is reasonable for transient failures, but retries without exponential backoff and jitter can turn a brief dependency hiccup into a self-inflicted traffic spike (a "retry storm") that prevents the dependency from recovering, since every failed caller retries in near-lockstep and compounds the load exactly when the dependency is least able to absorb it.

Fallback Behavior for Non-Critical Dependencies

A non-critical dependency's failure — a recommendation service, a personalization feature — should degrade the user experience gracefully (a default response, a cached value, a hidden feature) rather than failing the entire request. Distinguishing which dependencies are truly critical to a request's success versus enhancing it is a deliberate design decision, not an accident of how the code happened to be written.

Voluntary Disruption Management

Pod Disruption Budgets

As covered under deployment safety guidelines, PodDisruptionBudgets bound how much capacity can be removed simultaneously during voluntary disruptions (node drains, cluster upgrades), which is what allows routine cluster maintenance to proceed without becoming an availability incident for the workloads running on the affected nodes.

Graceful Shutdown Handling

A workload that doesn't handle SIGTERM correctly — finishing in-flight requests, deregistering from load balancing, closing connections cleanly within terminationGracePeriodSeconds — turns every routine Pod termination (scale-down, rolling update, node drain) into a source of dropped requests, even though the platform gave it every opportunity to shut down cleanly.

Testing Reliability Assumptions

Chaos Engineering

Deliberately injecting failure — killing Pods, introducing network latency, draining nodes — in a controlled way (via tools built for this purpose) validates that the redundancy, health checks, and graceful degradation designed into a system actually behave as expected under real failure conditions, rather than only being validated in the reviewer's head at design time.

Game Days and Failure Drills

Periodically rehearsing failure scenarios (a zone outage, a database failover) with the team that would actually respond to them surfaces gaps in runbooks, alerting, and tooling long before a real incident does, when the cost of discovering those gaps is a scheduled exercise rather than an unplanned outage.

Capacity and Load Reliability

Headroom for Failure, Not Just Peak Load

Capacity planning should account for the load increase that occurs when a portion of the fleet fails — if the system runs at full utilization across all replicas during normal peak load, losing even one replica during that peak overloads the survivors, turning a partial failure into a cascading one.

Load Shedding

For systems that can be pushed past their sustainable capacity by unexpected demand, explicit load shedding (rejecting excess requests deliberately, ideally with clear signaling to the caller) preserves the system's ability to serve the requests it can handle, rather than degrading uniformly until every request — including ones that would otherwise have succeeded — times out.

Example Configuration

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: codartium-api-pdb
spec:
  minAvailable: "75%"
  selector:
    matchLabels:
      app: codartium-api
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: codartium-api
spec:
  replicas: 6
  template:
    spec:
      terminationGracePeriodSeconds: 45
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: codartium-api
              topologyKey: "topology.kubernetes.io/zone"

Practical Consequences

Applying these guidelines produces systems that absorb routine node failures, zone disruptions, and dependency degradation without user-visible impact, and that fail predictably and gracefully at the margins when genuinely overwhelmed. Neglecting them commonly produces outages triggered by entirely routine events — a scheduled node drain, a brief downstream latency spike — that a correctly designed system would have absorbed without anyone noticing, precisely because the redundancy and graceful-degradation behavior needed to absorb them was never actually built in.