15.1.1 Container Log Streams

Container log streams refer to the actual stdout and stderr file descriptors of a container's PID 1 process, the mechanism by which Docker captures them at the daemon level, and the specific behaviors, multiplexing, buffering, and the limitations that arise when more than one process inside a container tries to produce log output through the same captured streams.

How Docker captures the streams

When a container starts, Docker attaches to the stdout and stderr file descriptors of the container's main process and redirects that output to whichever logging driver is configured, writing it to a file, a socket, or a remote endpoint depending on the driver:

docker run -d --name my-api my-api:1.4.0
docker logs my-api

This capture happens at the container runtime level, independent of whether the application inside the container is aware it is running inside a container at all; as long as it writes to the standard streams, Docker captures the output.

Multiplexing stdout and stderr

Docker's log storage and the docker logs API multiplex stdout and stderr into a single stream internally, tagging each chunk of output with which stream it originated from, which is how docker logs is able to optionally filter or color-distinguish error output from standard output even though both are interleaved in the order they were written:

docker logs my-api 2>/dev/null

docker logs my-api 1>/dev/null

Redirecting one of the two streams away when consuming docker logs output from the shell demonstrates that Docker has preserved the distinction between the two, even though they were captured together as the container ran.

Buffering behavior and its effect on log timing

Whether log output appears immediately or with a delay in docker logs -f depends partly on how the application itself buffers its own output, not just on Docker's capture mechanism. Many language runtimes buffer stdout when it is not connected to an interactive terminal, which is exactly the situation a containerized process runs in by default:

import sys
print("This may be delayed due to buffering", flush=False)

print("This appears immediately", flush=True)

ENV PYTHONUNBUFFERED=1

Setting an unbuffered mode at the language runtime level, where supported, ensures log output reaches Docker's capture mechanism (and therefore any downstream aggregation) as soon as it is written, rather than sitting in an application-level buffer until that buffer fills or the process exits.

The challenge of multiple processes writing to the same streams

Docker's stream capture is designed around a single main process per container; if that process spawns child processes that also write to stdout or stderr, their output is captured as well, since child processes typically inherit the same file descriptors, but distinguishing which process produced which line becomes harder without the application itself tagging its own output:

const { spawn } = require('child_process');
const worker = spawn('worker-script.sh', { stdio: 'inherit' });

console.log(JSON.stringify({ process: 'main', message: 'Starting worker' }));

Including a process or component identifier directly in structured log output, rather than relying on Docker's capture mechanism to distinguish sources, is the practical way to keep multi-process container logs interpretable once aggregated.

Why running multiple unrelated processes complicates log streams

A container running several genuinely independent processes, rather than one main process with occasional child processes, makes log stream capture considerably messier, since Docker's capture is tied to the container's PID 1 and does not automatically distinguish or separately manage streams for unrelated sibling processes the way separate containers would:

CMD ["sh", "-c", "service-a & service-b & wait"]

This pattern, running two unrelated services inside one container via a shell script, is generally discouraged specifically because of how it degrades log stream clarity: both services' output is merged into the same captured stream with no inherent way to attribute a given line to the correct service, which is one of several reasons the one-process-per-container convention is preferred in practice.

Following streams in real time

The -f (follow) flag on docker logs keeps the connection to the daemon's log stream open and continues printing new output as it is written, functioning similarly to tail -f against a regular file:

docker logs -f --tail 50 my-api

docker compose logs -f api db

Following multiple containers' streams simultaneously through Compose interleaves their output with a service name prefix, which is convenient for quick, multi-service debugging directly in a terminal, though it is still not a substitute for a proper aggregation system once the number of services or the retention requirement grows beyond what fits comfortably in a terminal scrollback.

Attach versus logs

docker attach connects directly and interactively to a container's stdin, stdout, and stderr, which is a different operation from docker logs: attach provides a live, two-way connection (and can be used to send input to the container's process), while logs only reads the captured output the logging driver has stored:

docker attach my-api

Detaching from an attached session without stopping the container requires the correct key sequence (Ctrl+P, Ctrl+Q by default); an attach session is generally not the right tool for routine log inspection, since accidentally sending input or signals to the container's main process is a real risk, and docker logs is almost always the safer, read-only choice for that purpose.

Common mistakes

• Running multiple unrelated processes in one container via a shell script, merging their log output into a single, hard-to-attribute stream.
• Leaving a language runtime's default output buffering in place, causing log output to appear with unexpected delay or only in large, infrequent bursts.
• Using docker attach for routine log inspection instead of docker logs, risking accidental interaction with the container's main process.
• Assuming docker logs provides cross-host visibility, when it only reads from the local daemon's captured stream for containers it manages directly.
• Failing to include a process or component identifier in log output from a container that does run more than one logical source of output, making aggregated logs difficult to attribute correctly.

Understanding container log streams as Docker's direct capture of a single main process's stdout and stderr, multiplexed but undistinguished by source beyond that, clarifies both why the one-process-per-container convention matters for log clarity and why application-level buffering and structured tagging are necessary complements to what Docker's capture mechanism provides on its own.