Logs, Shells, and Exec: Practical Techniques for Debugging Running Docker Containers

Debugging a running container is mostly about answering a few concrete questions quickly:

• Is the process running and healthy?
• What is it doing right now (and why)?
• What does it see (filesystem, env vars, network, DNS, mounted volumes)?
• What is it emitting (logs, metrics, errors)?
• Can I reproduce the problem from inside the same runtime context?

This tutorial focuses on practical, repeatable techniques using logs, shell access, and docker exec, plus a set of supporting commands that make those three tools far more effective.

Table of Contents

• Prerequisites and Setup
• Mental Model: Container vs Image vs Process
• Fast Triage Checklist
• 1) Debugging with docker logs
  • Understand Where Logs Come From
  • Essential docker logs patterns
  • Timestamps, tailing, and filtering
  • Common log pitfalls
• 2) Getting a Shell: Interactive Debugging
  • The difference between docker exec and docker attach
  • Choosing a shell (sh vs bash)
  • Working as root vs non-root
  • What to do when there is no shell
• 3) docker exec: Precision Debugging Without Restarting
  • Inspect processes, ports, and resources
  • Check filesystem, mounts, and permissions
  • Check environment variables safely
  • Network debugging from inside the container
  • DNS debugging
  • TLS/cert debugging
• 4) Supporting Commands That Multiply Your Effectiveness
  • docker ps, docker inspect, docker top
  • docker stats and resource pressure
  • Copy files in/out with docker cp
  • View filesystem changes with docker diff
  • Events timeline with docker events
• 5) Debugging Patterns and Real Scenarios
  • Scenario A: Container restarts in a loop
  • Scenario B: App is running but not reachable
  • Scenario C: “Works on my machine” env mismatch
  • Scenario D: Permission denied on a mounted volume
  • Scenario E: DNS failures inside container
• 6) Debugging Without Polluting Production
• 7) Quick Reference Command Cheat Sheet

Prerequisites and Setup

You should have:

• Docker Engine installed (docker version)
• Permission to run Docker commands (often via the docker group on Linux)
• A running container to debug

To have something concrete, you can run a simple web container:

docker run -d --name web -p 8080:80 nginx:alpine

And a container that intentionally fails:

docker run -d --name crashy alpine:3.19 sh -c 'echo starting; sleep 1; exit 1'

Mental Model: Container vs Image vs Process

A container is (simplifying slightly) a Linux process with:

• its own filesystem view (image layers + writable layer)
• its own network namespace (interfaces, routing, DNS config)
• its own process namespace (PID 1 inside container)
• optional resource limits (cgroups)

Most debugging boils down to: what is PID 1 doing, and what is the environment around it?

Fast Triage Checklist

When something is wrong, run this sequence first:

1. Is it running?

   docker ps --filter name=web
   docker ps -a --filter name=crashy

2. What does it say?

   docker logs --tail 200 crashy

3. Why did it exit?

   docker inspect crashy --format '{{.State.Status}} {{.State.ExitCode}} {{.State.Error}}'

4. If it’s running, what is it doing?

   docker top web
   docker stats --no-stream web

5. If needed, enter it (without restarting):

   docker exec -it web sh

1) Debugging with docker logs

Understand Where Logs Come From

By default, Docker captures whatever the container writes to:

• stdout (standard output)
• stderr (standard error)

This is why “best practice” for containers is to log to stdout/stderr rather than writing log files inside the container.

Docker stores and serves these logs via a logging driver (commonly json-file locally). If your container logs to files only, docker logs may show nothing even though the app is “logging.”

Check the logging driver:

docker inspect web --format '{{.HostConfig.LogConfig.Type}}'

If you see none, docker logs will not help.

Essential docker logs patterns

Show all logs (can be large):

docker logs web

Tail the last N lines:

docker logs --tail 200 web

Follow logs like tail -f:

docker logs -f web

Follow with a tail baseline (very common during incidents):

docker logs -f --tail 200 web

Timestamps, tailing, and filtering

Add timestamps:

docker logs -f --timestamps web

Show logs since a specific time:

docker logs --since 10m web
docker logs --since "2026-03-04T10:00:00" web

Show logs until a time:

docker logs --until 5m web

Combine --since with --tail to reduce noise:

docker logs --since 30m --tail 300 web

If you need filtering, pipe through tools you already know:

docker logs --since 1h web | grep -i error
docker logs --since 1h web | sed -n '1,120p'

Common log pitfalls

1. The container restarts and logs “disappear”

   • docker logs shows logs for the current container instance. If a container is recreated (new container ID), you need the new one.
   • If it is restarting, logs can be interleaved. Check restart count:

   docker inspect crashy --format 'restarts={{.RestartCount}} status={{.State.Status}}'

2. App logs are in files

   • You may need to docker exec and inspect /var/log/... or app-specific paths.
   • Consider changing the app to log to stdout in the long term.

3. Log driver is not compatible with docker logs

   • Some centralized drivers don’t support reading back logs.
   • Confirm driver and consult its behavior.

2) Getting a Shell: Interactive Debugging

The difference between docker exec and docker attach

• docker exec starts a new process inside the existing container.

  • Best for debugging.
  • Doesn’t interfere with PID 1 (usually).
  • You can run commands, open a shell, inspect state.

• docker attach attaches your terminal to the container’s main process (PID 1).

  • Useful for interactive apps designed for it.
  • Risky: you can send signals (like Ctrl+C) to PID 1 and stop the container.

In most debugging cases, prefer:

docker exec -it <container> sh

Choosing a shell (sh vs bash)

Many minimal images (Alpine, distroless, scratch-based) do not include bash.

Try sh first:

docker exec -it web sh

If you know bash exists:

docker exec -it some-ubuntu-container bash

If you’re unsure what’s available, you can probe:

docker exec web ls -l /bin
docker exec web sh -lc 'command -v bash || echo "no bash"'

Working as root vs non-root

Containers often run as non-root for security. That can limit your ability to read files or install tools.

Check the user:

docker exec web id

Run a command as root (if permitted by image/container settings):

docker exec -u 0 -it web sh

Run as a specific user:

docker exec -u 1000:1000 -it web sh

If the container is configured with a read-only filesystem or dropped capabilities, even root may be limited. That’s expected in hardened environments.

What to do when there is no shell

Some production images are distroless and contain only the app binary. docker exec -it ... sh will fail.

Options:

1. Use a debug sidecar container sharing the network namespace

   • Use --network container:<name> to debug networking from another container:

   docker run --rm -it --network container:web nicolaka/netshoot sh

   This gives you tools like curl, dig, tcpdump without modifying the target container.

2. Use a debug container sharing the PID namespace (advanced)

   • If allowed, you can join PID namespace to inspect processes. This is more sensitive and not always permitted.

3. Rebuild image with a debug variant

   • Maintain myapp:prod and myapp:debug (with shell + tools).
   • Use debug image only in non-production or controlled environments.

3) docker exec: Precision Debugging Without Restarting

docker exec is the workhorse: run one command, get one answer.

General form:

docker exec [options] <container> <command> [args...]

Common options:

• -i keep STDIN open
• -t allocate a pseudo-TTY (interactive)
• -u run as user
• -e set environment variable for the exec’d process
• -w set working directory

Example:

docker exec -it -u 0 -w / web sh

Inspect processes, ports, and resources

List processes inside the container:

docker exec web ps aux

If ps is missing (minimal images), use Docker’s host-side view:

docker top web

Check listening ports inside container (depends on tools available):

docker exec web sh -lc 'ss -lntp || netstat -lntp'

If neither exists, you can still infer via application logs and docker port:

docker port web

Check open files (if lsof exists):

docker exec web lsof -p 1 | head

Check memory/CPU quickly:

docker stats --no-stream web

For a snapshot of cgroup limits from inside (paths vary by cgroup v1/v2):

docker exec web sh -lc 'cat /sys/fs/cgroup/memory.max 2>/dev/null || true; cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || true'

Check filesystem, mounts, and permissions

See mounts:

docker exec web mount | sed -n '1,120p'

Check disk usage:

docker exec web df -h
docker exec web du -sh /var/cache/nginx 2>/dev/null || true

Find recent changes (useful when you suspect runtime writes):

docker diff web

Inspect permissions on a mounted volume path:

docker exec web ls -ld /usr/share/nginx/html
docker exec web ls -l /usr/share/nginx/html | head

If your app writes to a volume and fails with permission errors, confirm UID/GID:

docker exec web sh -lc 'id; stat -c "%U %G %a %n" /usr/share/nginx/html'

Check environment variables safely

Environment variables often explain “it works locally but not in prod.”

List environment:

docker exec web env | sort

If secrets might be present, avoid dumping everything into logs. Instead, query specific keys:

docker exec web sh -lc 'printf "APP_ENV=%s\n" "$APP_ENV"'
docker exec web sh -lc 'printf "DATABASE_URL is %s\n" "${DATABASE_URL:+set}"'

See how PID 1 was launched:

docker inspect web --format 'cmd={{json .Config.Cmd}} entrypoint={{json .Config.Entrypoint}}'

Network debugging from inside the container

From inside, verify:

• IP address
• default route
• connectivity to dependencies
• proxy settings

Commands (depending on image tooling):

docker exec web ip addr
docker exec web ip route

Test HTTP connectivity:

docker exec web sh -lc 'wget -qO- http://127.0.0.1:80 | head'

If curl is available:

docker exec web curl -v http://127.0.0.1:80/

If neither exists, use a debug container sharing the network:

docker run --rm -it --network container:web nicolaka/netshoot curl -v http://127.0.0.1:80/

Check if the service listens on the expected interface. A classic bug is binding only to 127.0.0.1 inside the container, making it unreachable from outside.

Inside container:

docker exec web sh -lc 'ss -lntp 2>/dev/null || netstat -lntp 2>/dev/null'

If you see it listening on 127.0.0.1:PORT rather than 0.0.0.0:PORT, it won’t be reachable via published ports.

DNS debugging

DNS issues are extremely common: wrong resolvers, missing search domains, broken upstream.

Inside container:

docker exec web cat /etc/resolv.conf
docker exec web cat /etc/hosts

Test resolution (if getent exists):

docker exec web getent hosts example.com

If dig/nslookup not present, use netshoot:

docker run --rm -it --network container:web nicolaka/netshoot dig example.com
docker run --rm -it --network container:web nicolaka/netshoot nslookup example.com

Also check whether a corporate proxy or DNS policy is required. From inside, inspect proxy env vars:

docker exec web sh -lc 'env | grep -i proxy || true'

TLS/cert debugging

If your container calls HTTPS services and fails with certificate errors:

• confirm system time (bad time breaks TLS)
• confirm CA bundle exists
• reproduce with verbose client output

Check time:

docker exec web date -u

Check common CA bundle locations:

docker exec web sh -lc 'ls -l /etc/ssl/certs 2>/dev/null || true; ls -l /etc/ssl/cert.pem 2>/dev/null || true'

Reproduce with verbose TLS (using netshoot if needed):

docker run --rm -it --network container:web nicolaka/netshoot curl -vk https://example.com/

4) Supporting Commands That Multiply Your Effectiveness

docker ps, docker inspect, docker top

List containers:

docker ps
docker ps -a

Filter by name:

docker ps --filter name=web

Inspect key runtime details:

docker inspect web

Extract specific fields with Go templates:

docker inspect web --format 'ip={{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'
docker inspect web --format 'restart={{.HostConfig.RestartPolicy.Name}}'
docker inspect web --format 'image={{.Config.Image}}'

Show processes from the host perspective:

docker top web

This is useful when the container lacks ps.

docker stats and resource pressure

Resource problems often look like “random timeouts” or “it hangs.”

docker stats
docker stats --no-stream web

If memory is near limit, the kernel may kill processes (OOM). Check if the container was OOMKilled:

docker inspect web --format 'oomkilled={{.State.OOMKilled}} exit={{.State.ExitCode}}'

Copy files in/out with docker cp

When you need config files, generated artifacts, or logs:

Copy from container to host:

docker cp web:/etc/nginx/nginx.conf ./nginx.conf

Copy into container (be cautious in production):

docker cp ./index.html web:/usr/share/nginx/html/index.html

docker cp is also a clean way to extract a crash dump without installing tooling inside.

View filesystem changes with docker diff

docker diff shows what changed in the container’s writable layer since start:

docker diff web

Output uses codes:

• A added
• C changed
• D deleted

This helps detect unexpected writes (e.g., app writing to /etc, or logs filling /var).

Events timeline with docker events

When debugging restarts, health checks, or orchestration behavior, a timeline is invaluable:

docker events --since 30m

Filter to one container:

docker events --since 30m --filter container=web

You can see die, restart, health_status: unhealthy, etc.

5) Debugging Patterns and Real Scenarios

Scenario A: Container restarts in a loop

Symptoms:

• docker ps shows it up briefly then gone
• docker ps -a shows Restarting (1) ...

Steps:

1. Check status and exit code:

   docker ps -a --filter name=crashy
   docker inspect crashy --format 'status={{.State.Status}} exit={{.State.ExitCode}} error={{.State.Error}} finished={{.State.FinishedAt}}'

2. Read logs from the last run:

   docker logs --tail 200 crashy

3. If it exits too fast to exec, temporarily disable restart policy (if set) or run a one-off container with the same image/command to reproduce:

   docker inspect crashy --format '{{.HostConfig.RestartPolicy.Name}}'

   Re-run interactively:

   docker run --rm -it alpine:3.19 sh -lc 'echo starting; sleep 1; exit 1'

4. If it’s a real app, reproduce with the same env vars and mounts. docker inspect can show mounts and env.

Scenario B: App is running but not reachable

Symptoms:

• container is “Up”
• curl localhost:published_port on host fails

Steps:

1. Confirm port publishing:

   docker ps --filter name=web
   docker port web

2. Confirm the app listens on expected port inside:

   docker exec web sh -lc 'ss -lntp 2>/dev/null || netstat -lntp 2>/dev/null'

3. Test from inside container:

   docker exec web sh -lc 'wget -qO- http://127.0.0.1:80 | head'

4. If it listens only on 127.0.0.1, change app config to bind 0.0.0.0.

5. Confirm host-side connectivity:

   curl -v http://127.0.0.1:8080/

6. If still failing, look for firewall rules, wrong interface binding, or reverse proxy issues.

Scenario C: “Works on my machine” env mismatch

Symptoms:

• same image behaves differently across environments

Steps:

1. Compare env vars:

   docker exec web env | sort > /tmp/env.running

2. Compare with expected env (from compose, CI, or documentation). Also inspect image defaults:

   docker inspect web --format '{{json .Config.Env}}' | jq .

3. Check config files inside container:

   docker exec web ls -la /app /etc 2>/dev/null || true
   docker exec web sh -lc 'find / -maxdepth 3 -name "*.conf" 2>/dev/null | head -n 50'

4. Validate the app is reading the env you think it is (many frameworks log config at startup—check docker logs).

Scenario D: Permission denied on a mounted volume

Symptoms:

• logs show EACCES / permission denied
• only happens with bind mounts or named volumes

Steps:

1. Identify mounts:

   docker inspect web --format '{{json .Mounts}}' | jq .

2. Check ownership and permissions inside container:

   docker exec web sh -lc 'id; ls -ld /path/to/mount; stat -c "%u:%g %a %n" /path/to/mount'

3. If container runs as non-root, align host permissions with container UID/GID.

   • On Linux, you may need to chown the host directory to match the container user.
   • In Kubernetes, you might use fsGroup or securityContext; in Docker Compose, you might set user:.

4. Confirm the app path is writable:

   docker exec web sh -lc 'touch /path/to/mount/.write_test && rm /path/to/mount/.write_test'

Scenario E: DNS failures inside container

Symptoms:

• curl https://service fails with “could not resolve host”
• intermittent resolution failures

Steps:

1. Inspect resolver config:

   docker exec web cat /etc/resolv.conf

2. Try resolution:

   docker run --rm --network container:web nicolaka/netshoot dig service.internal

3. If using Docker’s embedded DNS (common with user-defined bridge networks), ensure the container is on the correct network:

   docker network ls
   docker inspect web --format '{{json .NetworkSettings.Networks}}' | jq .

4. Confirm the target service name exists on that network (for Docker Compose, service names become DNS names on the compose network).

6) Debugging Without Polluting Production

Interactive debugging is powerful but can be risky. A few guidelines:

• Prefer read-only commands first (cat, ps, ss, env).
• Avoid installing packages in a running production container; it changes state and complicates incident review.
• Prefer ephemeral debug containers (like netshoot) attached to the same network namespace:

  docker run --rm -it --network container:web nicolaka/netshoot sh

• If you must modify something, capture evidence first:
  • docker logs --since ...
  • docker inspect ...
  • docker cp configs/logs out
  • docker diff before/after

Also remember: containers are meant to be replaceable. The long-term fix is usually a new image/config, not a series of manual edits.

7) Quick Reference Command Cheat Sheet

Replace CONTAINER with your container name or ID.

Status and metadata

docker ps
docker ps -a
docker inspect CONTAINER
docker inspect CONTAINER --format '{{.State.Status}} {{.State.ExitCode}}'
docker top CONTAINER
docker stats --no-stream CONTAINER

Logs

docker logs CONTAINER
docker logs --tail 200 CONTAINER
docker logs -f --tail 200 CONTAINER
docker logs --since 10m CONTAINER
docker logs -f --timestamps CONTAINER

Exec and shell

docker exec CONTAINER whoami
docker exec -it CONTAINER sh
docker exec -u 0 -it CONTAINER sh
docker exec -it -w / CONTAINER sh

Files and changes

docker cp CONTAINER:/path/in/container ./localfile
docker cp ./localfile CONTAINER:/path/in/container
docker diff CONTAINER

Networking

docker port CONTAINER
docker exec CONTAINER ip addr
docker exec CONTAINER ip route
docker exec CONTAINER cat /etc/resolv.conf
docker run --rm -it --network container:CONTAINER nicolaka/netshoot curl -v http://127.0.0.1:PORT/
docker run --rm -it --network container:CONTAINER nicolaka/netshoot dig example.com

Timeline

docker events --since 30m
docker events --since 30m --filter container=CONTAINER

Closing Workflow: A Practical Debug Loop

A reliable debugging loop for running containers looks like this:

1. Observe: docker ps, docker stats, docker logs --since ...
2. Localize: docker inspect (ports, mounts, env, restart policy, healthcheck)
3. Interrogate: docker exec one-liners to test assumptions (processes, files, DNS, HTTP)
4. Reproduce: run the same request from inside (or from a netshoot container sharing network)
5. Fix properly: update image/config, redeploy, and verify with logs and health checks

If you want, share a specific docker ps line, docker inspect snippet (redact secrets), and the last ~50 lines of docker logs, and I can suggest a targeted command sequence for your case.