16.3.1.1 Container Layer Data

Container layer data is whatever a running container has written to its own writable layer, the thin, container-specific filesystem layer sitting on top of its image's read-only layers, and troubleshooting issues related to it requires understanding exactly how this layer behaves, how it differs from a volume, and what happens to its contents at each point in a container's lifecycle.

How the writable layer fits into the overall image structure

Every container gets its own writable layer when created, stacked on top of the read-only layers from its image using the storage driver's union filesystem mechanism, most commonly overlay2 on modern Linux systems:

docker inspect my-api --format '{{.GraphDriver.Data}}'

map[LowerDir:/var/lib/docker/overlay2/.../diff:... MergedDir:/var/lib/docker/overlay2/.../merged UpperDir:/var/lib/docker/overlay2/.../diff WorkDir:/var/lib/docker/overlay2/.../work]

The UpperDir shown here is the container's actual writable layer; any file the container creates, modifies, or deletes is recorded here, while the underlying LowerDir layers from the image remain untouched and shared, potentially, with every other container started from the same image.

Inspecting what a container has actually written

The size and content of a container's writable layer can be inspected directly, which is useful for understanding exactly how much a specific container's runtime activity has added beyond its base image:

docker ps -s

CONTAINER ID   SIZE
3f29a8c1d8e2   340MB (virtual 1.1GB)

The reported size here represents data specific to this container's own writable layer, separate from the (much larger, but shared across every container using the same image) virtual size that includes the underlying image layers as well.

Why writable layer content disappears on container removal

The writable layer is deleted entirely when its container is removed, which is precisely why anything written there and nowhere else does not survive a container being replaced, the exact mechanism behind a large share of data loss incidents involving applications that were never configured to use a volume:

docker exec my-api sh -c "echo 'test' > /tmp/marker.txt"
docker rm -f my-api
docker run -d --name my-api my-api:1.4.0
docker exec my-api cat /tmp/marker.txt

cat: /tmp/marker.txt: No such file or directory

This is expected, correct behavior, not a bug, and the same outcome would occur for any path inside the container that was not specifically backed by a mounted volume, regardless of how important the data written there might have seemed at the time.

Copy-on-write behavior for modified files from the image

When a container modifies a file that originally came from one of its image's read-only layers, the storage driver copies that file into the writable layer before applying the modification, leaving the original, unmodified version intact in the underlying image layer:

docker exec my-api sh -c "echo 'modified' > /etc/hostname"

This copy-on-write behavior means a container's writable layer can grow specifically from modifying existing files, not just creating new ones, which is occasionally a source of unexpectedly large writable layer growth for a container that frequently modifies many small files originally provided by its base image.

Persisting writable layer content into a new image

Although generally discouraged as a primary workflow, docker commit can capture a container's current writable layer content into a new image layer, effectively making that runtime-accumulated state part of the image itself going forward:

docker commit my-api my-api:debug-snapshot

This is occasionally useful for capturing a specific debugging state for later investigation, but it is not an appropriate substitute for proper volume-based persistence in a production workflow, since it bakes a specific runtime snapshot into an image rather than maintaining genuinely live, ongoing persistent state the way a volume does.

Diagnosing unexpectedly large writable layers

A container whose writable layer has grown disproportionately large relative to what its application logic should be writing is worth investigating directly, since this often indicates logs, temporary files, or cache data accumulating somewhere that should instead be either rotated, cleaned up periodically, or moved to a properly managed volume:

docker exec my-api du -sh / --exclude=/proc --exclude=/sys 2>/dev/null | sort -rh | head -10

Running a directory size breakdown directly inside the running container identifies exactly which paths are responsible for the writable layer's growth, which usually points clearly toward either an unbounded log file, an accumulating temp directory, or a cache that was never configured with any eviction or size limit.

The interaction between writable layer growth and host disk usage

Because every running container's writable layer consumes actual disk space on the host, a fleet of containers each individually accumulating modest, unintended writable layer growth can collectively contribute to meaningful host-level disk pressure, which is worth checking specifically when host disk usage seems higher than the sum of image sizes alone would suggest:

docker ps -s --format "table {{.Names}}\t{{.Size}}"

Reviewing this across every running container on a host surfaces which specific ones are contributing disproportionately, which directs cleanup or configuration effort toward the actually responsible containers rather than a blanket host-wide cleanup that does not address the underlying, recurring cause.

Distinguishing writable layer issues from volume issues

When troubleshooting a storage-related problem, confirming whether the path in question is actually backed by a volume or sitting on the container's own writable layer determines which entirely different category of troubleshooting applies, persistence and backup considerations for a volume, versus growth, cleanup, and lifecycle considerations for the writable layer:

docker inspect my-api --format '{{json .Mounts}}'

A path appearing in this mounts list is backed by a volume or bind mount; any other path inside the container is sitting on the writable layer, subject to everything described above regarding its lifecycle and persistence characteristics.

Common mistakes

• Assuming all data written by a container is equally persistent, without checking which specific paths are actually backed by a volume versus sitting on the disposable writable layer.
• Not investigating unexpectedly large writable layer sizes, missing an accumulating log file, temp directory, or unbounded cache that should be redirected to a managed volume or given proper rotation and cleanup.
• Using docker commit as a substitute for proper volume-based persistence in a production workflow, rather than only as an occasional, deliberate debugging snapshot mechanism.
• Overlooking copy-on-write behavior as a contributor to writable layer growth, attributing growth solely to newly created files rather than also modified, originally image-provided ones.
• Not checking per-container writable layer sizes across a fleet when overall host disk usage seems unexpectedly high relative to the sum of image sizes alone.

Container layer data is, by design, disposable and tied entirely to its specific container's lifecycle, and understanding this clearly, what it is, how it grows, and exactly when it disappears, is what allows a clean distinction between data that genuinely needs volume-based persistence and data that is correctly, intentionally ephemeral and does not.