Kubernetes Ephemeral Storage Resource Management

Kubernetes Ephemeral Storage Resource Management is the practice of requesting, limiting, and monitoring the local, node-backed disk space a Pod consumes for its writable container layer, emptyDir volumes without an explicit size limit, and container logs — a resource dimension frequently overlooked relative to CPU and memory, but one whose exhaustion produces node-wide instability just as readily as memory pressure does. Ephemeral storage is called "ephemeral" specifically because it has no persistence guarantee: it exists only for the lifetime of the Pod and is reclaimed entirely once the Pod is deleted, in contrast to PersistentVolumeClaims, which survive Pod deletion and recreation.

Because ephemeral storage draws from the same underlying node disk that also holds the container runtime's image cache, kubelet state, and OS files, uncontrolled ephemeral storage consumption by one workload can degrade or destabilize an entire node, affecting every other Pod scheduled there.

What Counts as Ephemeral Storage

Container Writable Layers

Every container's writable filesystem layer (created by the container runtime on top of its read-only image layers) consumes ephemeral storage for any files the application writes directly into the container filesystem rather than to a mounted volume.

emptyDir Volumes Without a Medium Override

emptyDir volumes backed by node disk (the default, as opposed to medium: Memory, which uses RAM-backed tmpfs instead) count toward a Pod's ephemeral storage usage, since they are provisioned from the same node-local disk space.

Container Logs

Logs written by containers to stdout/stderr, captured and stored by the container runtime on the node's disk, also count toward ephemeral storage consumption, which is a detail worth remembering for verbose or high-throughput logging workloads.

Requesting and Limiting Ephemeral Storage

Declaring Requests and Limits

resources:
  requests:
    ephemeral-storage: "1Gi"
  limits:
    ephemeral-storage: "2Gi"

Requests inform the scheduler's fit calculation exactly like CPU and memory requests do, ensuring a node has sufficient free disk space reserved before the Pod is placed there; limits bound how much the Pod is allowed to actually consume before being evicted.

Enforcement Differs from CPU and Memory

Ephemeral storage limit enforcement is not a hard, immediate cgroup-level kill the way memory is — the kubelet periodically measures actual disk usage against the declared limit and evicts the Pod if it is found to be over, meaning there can be a delay between exceeding the limit and the Pod actually being terminated, unlike memory's immediate OOM kill.

Node-Level Disk Pressure

The DiskPressure Condition

When a node's available disk space (across the filesystem holding container images, writable layers, and logs) drops below a configured threshold, the kubelet reports a DiskPressure node condition and begins evicting Pods to reclaim space, similar in spirit to memory-pressure eviction but triggered by disk usage instead.

kubectl describe node node-worker-05 | grep -A 3 "DiskPressure"

Garbage Collection Before Eviction

Before resorting to Pod eviction, the kubelet first attempts garbage collection of unused container images and dead container filesystems, reclaiming space without disrupting running workloads — Pod eviction under DiskPressure is a later-stage response reserved for when garbage collection alone is insufficient to relieve the pressure.

Common Sources of Ephemeral Storage Exhaustion

Unbounded Log Growth

An application logging at excessive volume, without log rotation or a sensible verbosity level, can consume disk space unboundedly over a Pod's lifetime — a common, easily overlooked cause of DiskPressure that has nothing to do with the application's actual data processing needs.

emptyDir Used as Unintended Scratch Space

An emptyDir volume used for temporary file processing (downloading, transforming, and cleaning up files) without the application reliably cleaning up after itself accumulates disk usage over the Pod's lifetime, particularly for long-running Pods that never restart to reset the volume.

Missing sizeLimit on emptyDir

emptyDir.sizeLimit bounds how much space a specific emptyDir volume may consume, independent of the Pod's overall ephemeral storage limit — omitting it allows a single volume to grow without a dedicated ceiling, relying entirely on the Pod-level ephemeral storage limit (if set at all) to catch runaway growth.

volumes:
  - name: scratch
    emptyDir:
      sizeLimit: "500Mi"

Example

apiVersion: v1
kind: Pod
metadata:
  name: codartium-ephemeral-example
spec:
  containers:
    - name: app
      image: codartium/app:latest
      resources:
        requests:
          ephemeral-storage: "500Mi"
        limits:
          ephemeral-storage: "1Gi"
      volumeMounts:
        - name: scratch
          mountPath: /tmp/work
  volumes:
    - name: scratch
      emptyDir:
        sizeLimit: "300Mi"

kubectl describe pod codartium-ephemeral-example | grep -A 3 "Evicted"