20.3.2.3 Log Rotation Practice

Log rotation practice configures Docker's log driver to automatically limit the size of stored container log data, preventing a long-running container from filling the host filesystem with log output. Without rotation, the default json-file log driver writes unlimited data to disk. On a high-volume service that logs every request, this can consume gigabytes of disk space within days and eventually cause the host to run out of disk space, which affects all services on that host.

How Docker Stores Logs by Default

By default, Docker uses the json-file log driver. Each container's stdout and stderr output is appended to a JSON file at:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

Each log entry is a JSON object:

{"log":"2024-03-15T14:22:31.123Z GET /api/users 200 12ms\n","stream":"stdout","time":"2024-03-15T14:22:31.123456789Z"}

The file grows continuously. Docker does not rotate it unless explicitly configured to do so.

Finding Oversized Log Files

To identify containers with large log files on a Linux host:

find /var/lib/docker/containers -name "*-json.log" -exec ls -lh {} \; | sort -k5 -rh | head -20

Or using Docker's inspect to find the log path and then check size:

docker inspect my-api --format '{{.LogPath}}'

/var/lib/docker/containers/a1b2c3d4e5f6.../a1b2c3d4e5f6...-json.log

ls -lh /var/lib/docker/containers/a1b2c3d4e5f6.../*-json.log

-rw-r----- 1 root root 4.2G Mar 15 14:22 /var/lib/docker/containers/a1b2.../a1b2...-json.log

A 4.2GB log file from a single container is a disk space risk.

Configuring Log Rotation Per Container

Apply log rotation options when starting a container:

docker run -d \
  --name my-api \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=5 \
  my-api:latest

• max-size=10m — rotate the log file when it reaches 10MB.
• max-file=5 — keep at most 5 rotated log files.

Total maximum log storage for this container: 5 × 10MB = 50MB.

When the active log file reaches 10MB, Docker renames it to ...-json.log.1 and starts a new ...-json.log. When the next rotation occurs, ...-json.log.1 becomes ...-json.log.2. Files beyond the max-file count are deleted automatically.

Configuring Log Rotation Globally (Daemon Default)

Instead of configuring rotation on every container individually, set global defaults in the Docker daemon configuration:

Edit /etc/docker/daemon.json (create it if it does not exist):

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart the Docker daemon to apply:

systemctl restart docker

After this, all new containers use the configured rotation settings by default. Existing containers are not affected — they continue with whatever log configuration was set when they were created.

Per-container --log-opt settings override the daemon defaults for that specific container.

Log Rotation in Docker Compose

services:
  api:
    image: my-api:latest
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"

  db:
    image: postgres:15-alpine
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "3"

The logging section applies per-service log configuration. Services that produce more verbose logs (databases) can get a higher max-size; services with brief logs can use a smaller value.

Available Log Drivers

Docker supports multiple log drivers. The json-file driver supports rotation. Other drivers send logs to external systems:

When using a remote log driver (syslog, fluentd, awslogs), the container's output is not stored locally on the host at all — it goes directly to the remote system. This eliminates the local disk consumption problem entirely.

The none Driver

For containers that produce high-volume diagnostic logs that are not needed (CI runner containers, build containers), the none driver discards all output:

docker run -d \
  --log-driver none \
  my-build-tool

This means docker logs produces no output for this container. Use none only when log data is genuinely not needed.

Checking Current Log Configuration

docker inspect my-api --format '{{json .HostConfig.LogConfig}}'

{"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}

If the output shows {"Type":"json-file","Config":{}}, no rotation is configured and the log file grows without limit.

Emergency Log Truncation

If a log file has already grown very large and must be cleared immediately without stopping the container, the log file can be truncated on Linux:

truncate -s 0 /var/lib/docker/containers/<container-id>/*-json.log

This empties the file without removing it (the running Docker process keeps the file descriptor open). The file continues to grow from the truncated point. This is a one-time emergency measure — configure rotation to prevent recurrence.

Sizing Rotation Settings for Production

The appropriate max-size depends on:

• Log volume: A service logging every HTTP request at 100 req/s with 200-byte log entries produces ~20KB/s. At this rate, a 10MB file fills in ~8 minutes; 5 files cover ~40 minutes of history. A service with 1 req/s keeps 50MB of history for many hours.
• Debugging needs: Keep enough history to diagnose incidents that may not be detected immediately. 50–200MB total log storage per container is a common target.
• Disk capacity: On a host with many containers, multiply per-container storage by container count to estimate total disk commitment.

Typical production settings:

• High-traffic API: max-size=10m, max-file=10 → 100MB maximum
• Database: max-size=20m, max-file=5 → 100MB maximum
• Worker process: max-size=5m, max-file=5 → 25MB maximum