Kubernetes Health Signal Reliability

Kubernetes Health Signal Reliability is the practice of ensuring the probe mechanism itself, the thing generating liveness and readiness signals, is trustworthy: correctly tuned timing parameters, an appropriate probe type for the workload, and a probe check design that neither produces false positives under transient load nor false negatives that mask a genuinely unhealthy state.

Probe Types and Their Trade-offs

httpGet, tcpSocket, exec, and grpc

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080

livenessProbe:
  tcpSocket:
    port: 5432

livenessProbe:
  exec:
    command: ["pg_isready", "-U", "postgres"]

livenessProbe:
  grpc:
    port: 9090

httpGet is appropriate for any HTTP-serving workload and can return a rich status code distinguishing healthy from degraded; tcpSocket only confirms a port accepts connections, a weaker signal suitable when no application-level health endpoint exists; exec runs an arbitrary command inside the container, offering maximum flexibility at the cost of higher overhead per check; grpc, using the standard gRPC health checking protocol, is the natural choice for gRPC services without requiring a separate HTTP endpoint alongside the gRPC one.

Timing Parameter Tuning

The Five Core Parameters

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3
  successThreshold: 1

initialDelaySeconds delays the first check to accommodate normal startup time; periodSeconds sets check frequency; timeoutSeconds bounds how long a single check may take before being considered failed; failureThreshold requires consecutive failures before acting, absorbing transient blips; successThreshold (meaningful mainly for readiness) requires consecutive successes before transitioning back to healthy, preventing a single lucky check from prematurely restoring traffic to a still-recovering pod.

The Startup Probe Solution to the Timing Conflict

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 2
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

Before startup probes existed, a workload with highly variable startup time (sometimes 5 seconds, sometimes 90) forced a choice between a long initialDelaySeconds that wasted time on the common fast case, or a short one that killed the container during a legitimately slow but eventual successful startup; a startupProbe resolves this by disabling liveness and readiness checking entirely until it succeeds, after which the normal, tightly tuned liveness parameters take over.

Avoiding False Positives Under Resource Contention

Probe Failures Caused by Noisy Neighbors

A probe timing out not because the application is unhealthy but because the node is under CPU or I/O pressure from an unrelated workload produces a false positive, an unnecessary restart of a perfectly healthy container; setting timeoutSeconds with enough margin above the application's actual typical response time, and setting appropriate CPU requests so the kubelet's scheduling priority protects the probe-serving process, both reduce this risk.

resources:
  requests:
    cpu: 250m

Avoiding False Negatives from Shallow Checks

A Health Endpoint That Always Returns 200

# anti-pattern: endpoint returns success unconditionally
@app.route("/healthz")
def healthz():
    return "OK", 200

A health endpoint that returns success regardless of the application's actual internal state is a false negative generator, reporting healthy even when the application cannot serve real requests, defeating the entire purpose of health probing; a meaningful check verifies at minimum that the application's core request-handling path is functional, distinct from the deeper dependency checks appropriate specifically for readiness rather than liveness.

Cost of Probing at Scale

Overhead Across Many Replicas

periodSeconds: 5, replicas: 200
-> 40 probe requests per second cluster-wide, per probe type

Aggressive probe frequency across a large number of replicas contributes measurable load to the application itself and, for exec probes specifically, to the node's process execution overhead; probe frequency should be tuned against the actual acceptable detection latency for that workload rather than defaulted to the shortest interval available without considering the cumulative cost across every replica.

The gRPC Health Checking Protocol

Standardized Health Reporting for gRPC Services

livenessProbe:
  grpc:
    port: 9090
    service: myapp.v1.HealthService

The gRPC probe type calls the standard grpc.health.v1.Health/Check RPC, which the application must implement following the gRPC health checking protocol specification, giving gRPC-native services a first-class probe mechanism equivalent to httpGet for HTTP services rather than requiring a workaround such as exposing a separate HTTP endpoint purely for health checking.

Relationship to Readiness Availability and Pod Recovery Basics

Health signal reliability is the meta-level concern underlying both readiness availability and pod recovery basics: those two areas describe what Kubernetes does in response to a probe's result, removing endpoint membership or restarting a container, while health signal reliability addresses whether that probe result itself can be trusted, since even perfectly implemented downstream reactions to probe failure provide no real reliability benefit if the probe generating the signal is poorly tuned, using the wrong check type, or producing false positives and negatives that misrepresent the workload's actual health.