6.19 Kubernetes Manifest Environment Variation

Kubernetes Manifest Environment Variation is the general problem, and the range of established solutions, for expressing configuration that must differ across deployment targets, such as development, staging, and production, while avoiding full duplication of otherwise identical manifest content, keeping the majority of a workload's configuration shared and centralizing only the genuinely environment-specific differences.

The Duplication Problem

Why Naive Copy-Paste Fails Over Time

The simplest approach to environment variation, maintaining a fully separate copy of every manifest per environment, works initially but degrades as the number of environments and objects grows, since any shared change, such as adding a new label convention, now needs to be manually replicated across every copy, a process prone to some copies being missed and quietly drifting out of sync.

What Should Vary Versus What Should Not

Environment-specific variation typically concentrates in a small, identifiable set of concerns: replica counts, resource requests and limits, image tags or digests, environment-specific ConfigMap and Secret values, and ingress hostnames, while the bulk of an object's structure, container definitions, volume mounts, and label conventions, usually remains identical across environments.

Templating as a Solution

Parameterizing With Template Variables

Templating tools, most notably Helm, address environment variation by treating manifests as templates with placeholder variables, and supplying a distinct set of values, often as a per-environment values file, at render time to produce the final, environment-specific manifest output.

Trade-Offs of Templating

Templating offers powerful parameterization, including conditionals and loops within the template logic itself, but this same flexibility can make the relationship between a rendered manifest and its source template harder to trace, and encourages logic to creep into what would otherwise be purely declarative configuration.

Overlays as a Solution

Patching a Common Base

Overlay-based tools, most notably Kustomize, take a different approach, maintaining one unmodified base set of manifests and layering small, targeted patches on top for each environment, expressing only the specific fields that differ rather than parameterizing the base manifest itself.

Trade-Offs of Overlays

Overlays keep the base manifests as plain, valid Kubernetes YAML with no template syntax, which some teams find easier to reason about and validate directly, though expressing highly conditional or deeply structural differences between environments can require more elaborate patch composition than an equivalent templated conditional would.

Environment-Specific Value Injection

ConfigMaps and Secrets as the Injection Point

Rather than varying a workload's core Deployment or Pod spec, many environment-specific differences are pushed down into ConfigMap and Secret content instead, letting the workload's own spec remain identical across environments while the actual configuration values it reads at runtime differ, keeping variation concentrated in a narrower, more auditable surface.

External Configuration Management Integration

Some environments source environment-specific values from external systems, such as a secrets manager or a centralized configuration service, injected into manifests at deployment time rather than stored directly in the manifest repository, further narrowing what needs to be manually duplicated or parameterized per environment within the manifests themselves.

Directory and Repository Organization Patterns

Environment-Segmented Directory Structures

A common convention organizes a repository with a shared base directory and per-environment directories, such as overlays/dev, overlays/staging, and overlays/production, each containing only the environment-specific patches or values needed on top of the shared base, making the repository's structure directly reflect the base-plus-variation model.

Branch-Based Versus Directory-Based Separation

Some teams instead use separate branches per environment rather than directories within a single branch, though directory-based separation within one branch is generally favored for GitOps workflows, since it allows a single commit to atomically express a coordinated change across multiple environments if needed, something branch-based separation makes considerably harder to achieve.

Choosing Between Approaches

Matching Tooling to Team Needs

The choice between templating and overlays, or a combination of both, generally comes down to how much genuine logic-driven variation exists versus how much is purely a matter of substituting specific values, with heavily logic-driven variation favoring templating's expressiveness and simpler, mostly value-based variation favoring the clarity of overlays applied to plain YAML.