Kubernetes Health Check Guidelines

Kubernetes Health Check Guidelines describe how a containerized application should expose its internal state to the kubelet so that the platform can make correct decisions about restarting, routing traffic to, and waiting on a container. Kubernetes does not infer health from process existence alone — a process can be running yet deadlocked, unable to reach a dependency, or still initializing — so it relies on probes that the application must implement deliberately and correctly.

The Three Probe Types

Liveness Probes

A liveness probe answers a single question: is this container in a state from which it can never recover on its own? If the probe fails repeatedly (per failureThreshold), the kubelet kills and restarts the container. Liveness probes should check only the internal health of the process itself — deadlocks, unrecoverable internal error states — and must never check the health of downstream dependencies such as a database or an external API, since a downstream outage would then trigger a restart loop of a container that was never actually broken.

Readiness Probes

A readiness probe answers whether the container is currently able to serve traffic. Failing the readiness probe removes the Pod's IP from the Endpoints object backing any Service that selects it, so traffic stops flowing to it, but the container is not restarted. Readiness probes are the correct place to check lightweight, essential dependencies — for example, whether a connection pool has been initialized — because temporarily removing an unready Pod from load balancing is the desired outcome, not a restart.

Startup Probes

A startup probe protects applications with long or variable initialization time. While a startup probe is defined and has not yet succeeded, liveness and readiness probes are disabled, preventing a slow-starting application from being killed by an impatient liveness probe before it has finished booting. Once the startup probe succeeds once, it is never checked again for the life of the container.

Probe Mechanisms

HTTP GET

The most common mechanism: the kubelet issues an HTTP GET request to a specified path and port, treating any response in the 200–399 range as success. Health endpoints implementing this should be cheap to compute and should not perform expensive work on every call.

TCP Socket

A TCP probe simply attempts to open a socket connection to the specified port. This is useful for protocols that don't speak HTTP, but only verifies that something is listening — not that the application logic behind that port is actually functional.

Exec

An exec probe runs a command inside the container and treats a zero exit code as success. This is flexible but incurs more overhead per check than HTTP or TCP probes, since it forks a process inside the container namespace each time.

gRPC

Native gRPC probing uses the standard gRPC health checking protocol, allowing services that only expose gRPC (no HTTP surface) to be probed without a sidecar or a custom exec script.

Timing Parameters

initialDelaySeconds

The number of seconds after container start before probing begins. This should reflect realistic startup time, though the startup probe is generally the more robust solution for highly variable startup durations.

periodSeconds

How frequently the probe is executed. Too frequent adds unnecessary load; too infrequent delays detection of a real problem.

timeoutSeconds

How long the kubelet waits for a probe response before considering it failed. This should exceed the p99 latency of the health endpoint under normal load to avoid false negatives during brief load spikes.

failureThreshold and successThreshold

failureThreshold is the number of consecutive failures required before the probe result flips to failed, giving tolerance for transient blips. successThreshold is the equivalent for recovery, and for liveness probes must be 1, since Kubernetes always treats a single liveness recovery as sufficient.

Designing Health Endpoints

Separation of Liveness and Readiness Endpoints

A well-designed application exposes distinct endpoints (e.g. /healthz/live and /healthz/ready) rather than a single shared health check, since the two probes have fundamentally different responsibilities and failure semantics.

Avoiding Cascading Failures

A readiness endpoint that checks a downstream dependency should degrade gracefully rather than immediately failing on the first hiccup — a brief retry or circuit-breaker pattern inside the health check logic prevents flapping between ready and not-ready states under minor downstream jitter.

Cheap and Deterministic Checks

Health endpoints should avoid heavy computation, external network calls with long timeouts, or side effects such as writes. A probe that is itself slow or resource-intensive can become a contributing cause of the very instability it's meant to detect.

Example Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: codartium-worker
spec:
  template:
    spec:
      containers:
        - name: codartium-worker
          image: registry.example.com/codartium-worker@sha256:aa11bb...
          startupProbe:
            httpGet:
              path: /healthz/startup
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
            successThreshold: 1

Practical Consequences

Correctly designed health checks let Kubernetes self-heal genuinely broken containers, avoid routing traffic to Pods that aren't ready, and tolerate slow startups without premature termination. Misconfigured health checks are a frequent source of two opposite failure modes: restart loops caused by liveness probes that check too much (including downstream dependencies), and prolonged outages caused by readiness probes that check too little, allowing broken Pods to keep receiving traffic.