Kubernetes Reliability Practice

Kubernetes Reliability Practice is the operational discipline of proactively verifying that the reliability mechanisms covered throughout this knowledge area actually work as intended before a real failure tests them, encompassing chaos engineering and game days, pre-production reliability review checklists, runbook preparation, and the postmortem process that feeds real incidents back into improved configuration.

Chaos Engineering and Game Days

Deliberately Inducing Failure to Verify Recovery

kubectl delete pod web-abc123 --grace-period=0 --force
kubectl cordon node-1 && kubectl drain node-1 --ignore-daemonsets

Deliberately terminating a pod, draining a node, or killing an entire availability zone's nodes in a controlled game-day exercise verifies that replica reliability, node failure tolerance, and zone failure tolerance actually behave as configured, surfacing gaps, a PodDisruptionBudget set too strictly, a topology spread constraint silently unsatisfied, before an uncontrolled real failure discovers the same gap under pressure.

Tools for Structured Fault Injection

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
spec:
  action: pod-kill
  selector:
    labelSelectors:
      app: web
  scheduler:
    cron: "@every 1h"

Fault injection tools such as Chaos Mesh or Litmus allow chaos experiments to be defined declaratively and run on a repeatable schedule, turning what would otherwise be a one-off manual exercise into an ongoing, automated verification practice that continues to catch reliability regressions as configuration evolves over time.

Pre-Production Reliability Review

A Checklist Before First Production Traffic

[ ] replicas >= 2 with N+1 margin
[ ] readiness and liveness probes configured and dependency-aware
[ ] PodDisruptionBudget consistent with replica count and HPA minReplicas
[ ] topology spread constraints across zones
[ ] resource requests set from observed usage; QoS class intentional
[ ] preStop hook and terminationGracePeriodSeconds tuned

A standard checklist applied before any workload receives production traffic operationalizes every mechanism covered throughout this knowledge area into a concrete gate, ensuring reliability configuration is a deliberate design decision made before launch rather than an afterthought discovered only after an incident reveals its absence.

Runbooks for Known Failure Scenarios

Documenting the Response Before It Is Needed

## Runbook: CrashLoopBackOff on web deployment
1. kubectl logs <pod> --previous
2. Check exit code: kubectl get pod <pod> -o jsonpath='...'
3. If OOMKilled: check memory limit vs actual usage trend
4. If application error: check recent deploy/config change

A runbook prepared in advance for common, previously-encountered failure patterns, CrashLoopBackOff, a stuck rollout, a stalled node drain, reduces mean time to recovery during an actual incident by replacing in-the-moment diagnosis with a known, tested sequence of checks, directly operationalizing the reliability model's emphasis on minimizing recovery time.

Postmortems Feeding Back Into Configuration

Blameless Analysis of Real Incidents

## Postmortem: Zone outage, 2024-06-01
- Impact: 15 minutes of 20% error rate
- Root cause: topology spread constraint set to ScheduleAnyway,
  allowing accidental concentration in one zone
- Action item: change to DoNotSchedule with minDomains: 3

A blameless postmortem process that identifies specific configuration gaps, not merely "human error", and converts each finding into a concrete follow-up change (tightening a spread constraint, adjusting a disruption budget, adding a missing runbook) is what turns each real incident into a durable improvement to the reliability mechanisms covered throughout this knowledge area, rather than a repeated, unaddressed recurring risk.

Capacity Planning as an Ongoing Practice

Revisiting Sizing and Redundancy Margins Periodically

Resource requests, replica floors, and disruption budgets sized correctly at launch drift out of alignment with reality as traffic patterns, team size, and infrastructure change; a recurring cadence of revisiting these values against actual observed usage (the same metrics covered under availability observation basics) keeps reliability configuration matched to current, not historical, operating conditions.

Relationship to Availability Observation Basics and the Broader Reliability and Availability Area

Reliability practice is the operational discipline that closes the loop across every mechanism covered throughout this knowledge area: where the specific reliability mechanisms, probes, disruption budgets, topology spread, replica floors, define what should happen under failure, and availability observation basics defines how to measure whether it is happening, reliability practice is the ongoing process, chaos testing, review gates, runbooks, and postmortems, that verifies those mechanisms continue to work correctly over the lifetime of a workload, not merely at the moment they were first configured.