6.21 Kubernetes Declarative Drift

Kubernetes Declarative Drift is the condition in which a cluster's actual, live object state diverges from what the authoritative source manifests declare, arising whenever changes are made directly against the cluster outside the normal manifest-and-apply workflow, undermining the core premise that manifests serve as a complete, trustworthy record of intended configuration.

How Drift Occurs

Manual Interventions

The most common source of drift is direct manual modification of a live object, such as an operator using kubectl edit or kubectl scale during an incident to quickly resolve a problem, changes that are never reflected back into the corresponding manifest and therefore silently diverge from what the manifest still declares.

Imperative Debugging Commands

Commands like kubectl set image or kubectl patch, used for quick, targeted fixes without going through a manifest change, similarly introduce drift, since they modify the live object directly while leaving the manifest, and any repository it lives in, unaware of the change.

Automated External Actors

Drift can also originate from automated systems outside the manifest workflow, such as an autoscaler adjusting replica counts, or a separate imperative tool managing certain fields directly, some of which is expected and benign drift, distinguished from drift representing a genuine loss of manifest-cluster synchronization.

Consequences of Unmanaged Drift

Manifests Become Untrustworthy

Once drift accumulates, the manifest repository, rather than reflecting the cluster's actual current state, only reflects an outdated or partial approximation of it, undermining the value of manifests as a reliable historical and reviewable record of what should be running.

Surprising Reversion on Reapplication

If a manifest is later reapplied, whether deliberately or through an automated pipeline's normal cadence, any drifted field the manifest still manages gets reverted back to the manifest's declared value, which can be surprising or disruptive if the drift had been an intentional, still-needed fix that was never backported into the manifest itself.

Erosion of Confidence in Automation

Repeated surprising reversions caused by drift being silently overwritten tend to erode operator trust in the manifest-driven workflow itself, sometimes leading teams to avoid reapplying manifests out of fear of undoing manual fixes, which further compounds the underlying drift problem over time.

Detecting Drift

Comparing Live State Against Source

Drift detection fundamentally means comparing the live cluster's current object state against what the source manifests declare, restricted to the fields the manifests actually manage, surfacing any mismatch as a candidate drift finding requiring investigation.

GitOps Tooling as Continuous Drift Detectors

GitOps reconciliation agents inherently perform continuous drift detection as part of their normal operation, since every reconciliation cycle involves comparing live state against the Git-declared source and, depending on configuration, either automatically correcting drift or surfacing it as an alert for human review.

Server-Side Dry Run as a Point-in-Time Check

Outside of a continuously running GitOps agent, a manual or scheduled kubectl diff-style check against the current manifest source provides a point-in-time drift detection capability, useful for periodic audits even without adopting full continuous reconciliation.

Managing and Preventing Drift

Backporting Necessary Manual Changes

When a manual change made during an incident turns out to be a legitimate, lasting fix rather than a temporary workaround, promptly updating the corresponding manifest to reflect that change closes the drift and restores the manifest as an accurate source of truth going forward.

Restricting Direct Cluster Write Access

Some teams reduce drift risk structurally, by limiting direct write access to production clusters and routing essentially all changes exclusively through the manifest-and-pipeline workflow, making manual, undocumented drift much rarer by removing the easy path to introduce it in the first place.

Accepting Deliberately Unmanaged Fields

For fields genuinely intended to be managed outside the manifest, such as replica counts under active autoscaling, deliberately excluding them from the manifest entirely, rather than declaring and then having them drift, avoids false-positive drift findings and aligns the manifest's declared scope with its actual intended responsibility.