15.2.1.4 Container Block IO

Container block I/O is the volume of read and write operations a container performs against block storage devices, measured by Docker through the kernel's cgroups blkio (or io, on newer cgroups v2 systems) accounting, providing per-container visibility into disk activity that is essential for diagnosing storage-related performance problems that CPU and memory metrics alone cannot explain.

Reading basic block I/O

Docker's stats interface reports cumulative bytes read from and written to block devices since the container started:

docker stats my-api --no-stream

CONTAINER ID   NAME     BLOCK I/O
3f29a8c1d8e2   my-api   45MB / 1.2GB

Like network I/O, this is a cumulative total rather than a current rate, so deriving an actual throughput figure requires comparing two readings over a known time interval rather than treating the raw numbers as an instantaneous rate.

What contributes to block I/O for a container

Block I/O includes both the container's writable layer (any files written inside the container that are not part of a mounted volume) and any volumes or bind mounts attached to it, which means high block I/O does not automatically indicate a volume-related issue; it could equally originate from temporary files, logs, or cache data written directly into the container's own filesystem layer:

docker exec my-api du -sh /var/lib/docker/containers/*/  2>/dev/null
docker exec my-api mount | grep overlay

Distinguishing writable-layer I/O from volume I/O matters because the performance characteristics and durability implications differ: the writable layer typically uses the same storage driver and backing filesystem as the rest of the host's container storage, while a volume might be backed by entirely different, potentially faster or slower, underlying storage.

Per-device accounting

For containers with multiple mounted volumes potentially backed by different physical or virtual devices, cgroups accounting can break down I/O per device, which is more precise than the single combined figure docker stats reports by default:

cat /sys/fs/cgroup/blkio/docker/<container-id>/blkio.throttle.io_service_bytes

8:0 Read 47185920
8:0 Write 1288490188
8:16 Read 0
8:16 Write 0

This per-device breakdown is particularly useful when a container has both a fast, low-latency volume for active data and a slower, higher-capacity volume for archival or less frequently accessed data, since it reveals which specific device is actually responsible for I/O volume rather than only the combined total.

Throttling block I/O

Docker supports limiting both the rate (bytes per second) and the number of operations per second a container can perform against a specific block device, useful for preventing one container's heavy disk activity from starving others sharing the same underlying physical storage:

docker run -d --device-write-bps /dev/sda:10mb my-api

docker run -d --device-read-iops /dev/sda:500 my-api

These limits apply per-device and require knowing the specific device path backing the relevant storage, which on cloud infrastructure with abstracted block storage may require checking how that storage is actually mapped to a device path on the host before the limit can be applied correctly.

Diagnosing I/O-bound performance problems

A container experiencing degraded performance with normal-looking CPU and memory usage, but elevated latency on operations that touch disk, is a strong candidate for an I/O-bound bottleneck rather than a compute-bound one, and block I/O metrics combined with latency observations help confirm this:

docker exec my-api iostat -x 1 5

docker stats my-api --no-stream

A high I/O wait time inside the container, alongside otherwise unremarkable CPU and memory figures, points toward storage as the bottleneck, which calls for a different remediation, faster underlying storage, reducing unnecessary disk writes, or better caching, than a CPU or memory limit adjustment would address.

The impact of overlay filesystem performance

Because the writable layer typically sits on an overlay filesystem (most commonly OverlayFS), write-heavy workloads that target the container's own filesystem rather than a mounted volume can experience overhead specific to how that storage driver handles writes, particularly for workloads with many small, frequent file modifications:

docker info --format '{{.Driver}}'

Moving write-heavy paths to a dedicated volume, rather than leaving them on the container's writable layer, often improves performance specifically because volumes can bypass some of the overlay filesystem's additional indirection, depending on the storage driver and volume type in use.

Block I/O contention across containers on shared storage

On a host where multiple containers share the same underlying physical disk, even containers with no explicit I/O limits set can experience performance degradation caused entirely by another container's heavy disk activity, a noisy-neighbor problem analogous to CPU or memory contention but specific to storage:

docker stats --no-stream --format "{{.Name}}: {{.BlockIO}}"

Comparing block I/O across every container on a host at the moment a specific container's performance degrades is a reasonable first step in ruling out or confirming this kind of shared-resource contention before assuming the affected container's own configuration is at fault.

Exporting block I/O metrics for trend analysis

As with other resource metrics, point-in-time block I/O readings are far less useful than a tracked history, since storage-related performance issues often build gradually, a growing dataset causing increasingly larger reads, or a cache that has stopped being effective, rather than appearing suddenly:

rate(container_fs_writes_bytes_total{name="my-api"}[5m])

Tracking this rate over time, alongside application-level latency metrics for operations known to touch disk, surfaces the correlation between rising I/O volume and degrading latency well before it becomes severe enough to be reported as a user-facing incident.

Common mistakes

• Assuming all block I/O originates from mounted volumes, without checking whether writable-layer activity (logs, temp files, caches) is actually the larger contributor.
• Treating the cumulative byte counters as an instantaneous rate rather than deriving throughput from two readings over a known interval.
• Setting I/O throttling limits without first identifying the correct underlying device path, especially on cloud infrastructure with abstracted block storage.
• Diagnosing performance issues by CPU and memory metrics alone, missing an I/O-bound bottleneck that those two metrics would not reveal.
• Not checking for noisy-neighbor contention from other containers sharing the same physical disk before assuming a specific container's own configuration is responsible for degraded performance.

Container block I/O metrics, read correctly alongside an understanding of what contributes to them, writable layer versus volumes, per-device breakdown, and throttling configuration, are essential for diagnosing storage-related performance problems that would otherwise be invisible to CPU and memory monitoring alone.