16.2.2.4 Container Stopped State

The container stopped state is where a container's main process has exited and is not currently running, and effectively troubleshooting from this state means understanding that a stopped container still retains valuable forensic information, its filesystem, its logs, and its final exit details, all of which remain inspectable even though nothing inside it is currently executing.

What persists after a container stops

Unlike a removed container, a merely stopped one retains its entire writable layer, its configuration, and the full history of its logs, none of which are lost simply because the process inside it exited:

docker ps -a

CONTAINER ID   STATUS
3f29a8c1d8e2   Exited (1) 5 minutes ago

docker logs my-api
docker inspect my-api

Both of these commands continue to work normally against a stopped container, which is why investigating a failure should generally happen before removing the container, since removal discards all of this retained, inspectable state permanently.

Inspecting the stopped container's filesystem

A stopped container's filesystem can be examined directly without restarting it, which is useful for confirming the state of files at the exact moment the process stopped, without any risk of that investigation itself disturbing or changing anything:

docker cp my-api:/app/logs/error.log ./error.log

docker export my-api -o my-api-snapshot.tar

Copying specific files out of the stopped container, or exporting its entire filesystem as a tarball, allows offline investigation of exactly what state the container was in when it stopped, which can be especially useful when the cause is suspected to involve a corrupted file or an unexpected filesystem state at the time of failure.

Restarting a stopped container versus creating a new one

docker start resumes a stopped container using its existing filesystem and configuration exactly as it was, which is meaningfully different from docker run, which always creates an entirely new container from the image:

docker start my-api

docker run -d --name my-api my-api:1.4.0

Restarting the existing, stopped container preserves whatever state existed in its writable layer at the time it stopped, which is useful when that state itself is relevant to understanding or reproducing the original issue, while running a fresh container discards that state entirely and starts clean.

Distinguishing a deliberate stop from a crash

A container's stop reason matters considerably for how it should be investigated, and the exit code combined with whether a stop was explicitly requested distinguishes a deliberate, expected stop from an unexpected crash:

docker inspect my-api --format '{{.State.ExitCode}} {{.State.Error}}'

0    (clean, possibly deliberate, exit)
137  (likely killed, possibly OOM or a stop timeout)
1    (an application-level error)

An exit code of 0 following a deliberate docker stop command is expected and requires no further investigation; the same exit code occurring unexpectedly, with no corresponding stop command issued by anyone, is a different situation worth understanding, since it suggests the application itself decided to exit for some reason that is worth identifying.

Investigating a container stopped due to a restart policy giving up

A container with a restart policy that includes a maximum retry count, or one using the default policy with no automatic restart at all, can end up in a stopped state after repeated failed restart attempts were exhausted, which is worth distinguishing from a single, first-time stop:

docker inspect my-api --format '{{.RestartCount}}'

A restart count significantly above zero, combined with a currently stopped state, indicates the container tried and failed to recover multiple times before ultimately remaining stopped, which points toward a persistent, non-transient cause rather than a single, isolated incident.

Comparing a stopped container's configuration against a working one

When a specific container instance stopped unexpectedly while sibling replicas of the same service continue running normally, comparing the stopped instance's configuration and final state against a still-running sibling often reveals what was actually different about the one that failed:

docker inspect my-api-failed --format '{{json .Config}}' > failed-config.json
docker inspect my-api-2 --format '{{json .Config}}' > healthy-config.json
diff failed-config.json healthy-config.json

A configuration difference surfaced this way, an environment variable, a resource limit, a mounted volume, often explains an otherwise puzzling situation where ostensibly identical containers experienced different outcomes.

Cleaning up after investigation

Once a stopped container has been fully investigated and any needed forensic information extracted, removing it is appropriate to avoid an accumulation of stopped containers consuming disk space unnecessarily, though this should happen deliberately, after investigation, rather than automatically or prematurely:

docker rm my-api

docker container prune --filter "until=24h"

Pruning stopped containers older than a certain age, rather than immediately and indiscriminately, provides a reasonable window during which a stopped container remains available for investigation before being cleaned up as part of routine host maintenance.

Common mistakes

• Removing a stopped container immediately after noticing it, before extracting any of the forensic information, logs, exit code, filesystem state, that would have been useful for understanding why it stopped.
• Using docker run to create a fresh container when docker start on the existing, stopped one would have preserved relevant state needed for the investigation.
• Treating every stopped container with the same urgency, without first distinguishing a deliberate, expected stop from a genuine, unexpected crash.
• Not checking restart count before investigating, missing the distinction between a first-time failure and one that occurred after several automatic retry attempts were already exhausted.
• Investigating a stopped container in isolation without comparing it against a still-running sibling replica when one is available.

A container in the stopped state is not a dead end for investigation; its retained logs, configuration, and filesystem remain fully inspectable, and working through exit code, restart history, and comparison against a healthy sibling, before removing it, extracts the maximum diagnostic value from a state that is often the most informative point in the entire troubleshooting process.