Stateful Containers Done Right: Debugging Data Loss and Volume Mount Mistakes

Stateful containers fail in surprisingly predictable ways. Most “mysterious data loss” incidents boil down to one of these root causes:

• You wrote data into the container filesystem (ephemeral layer) instead of a persistent volume.
• You mounted the wrong path (or mounted over the path that already had the data).
• You used an anonymous volume when you thought it was named (or vice versa).
• You rebuilt/recreated containers and assumed the data would follow automatically.
• You ran multiple containers against the same storage without understanding locking/consistency.
• You changed UID/GID or permissions and the app silently fell back to a different directory.

This tutorial focuses on how to do stateful containers correctly and, more importantly, how to debug and prove what’s happening using real commands. Examples use Docker, but the core principles apply to any container runtime.

Table of Contents

• 1. The mental model: container layer vs volumes
• 2. The most common data-loss patterns
• 3. Inspecting mounts and proving where data is going
• 4. Reproducing “data loss” on purpose (so you recognize it)
• 5. Mounting correctly: bind mounts vs named volumes
• 6. The “mount hides existing data” trap
• 7. Debugging with docker exec, findmnt, and stat
• 8. Anonymous volumes and why they confuse everyone
• 9. Permissions, UID/GID mismatches, and silent fallback directories
• 10. Backups, migrations, and safe recovery steps
• 11. Practical checklists

1. The mental model: container layer vs volumes

A running container sees a filesystem that is composed of:

1. Image layers (read-only)
2. Container writable layer (read-write, ephemeral)
3. Mounts (volumes, bind mounts, tmpfs) that appear at specific paths

If your application writes to a path that is not backed by a persistent mount, it writes into the container’s writable layer. That data disappears when:

• the container is removed (docker rm)
• the container is recreated (docker compose up --force-recreate)
• you deploy a new container instance and the old one is gone

Important nuance: Restarting a container (docker restart) does not remove the writable layer. Many people test persistence by restarting and conclude “it’s fine”, then lose data when they redeploy.

Quick proof: container layer is not a volume

Run a container, write a file, remove the container, and see it vanish:

docker run --name ephemeral-test -it --rm ubuntu:24.04 bash -lc 'echo hello > /tmp/hello.txt; ls -l /tmp/hello.txt; cat /tmp/hello.txt'

Because --rm removes the container at exit, the file is gone immediately after the container stops. If you omit --rm, the file persists only as long as that container object exists.

2. The most common data-loss patterns

Pattern A: Writing to the wrong directory

Your app writes to /var/lib/app but you mounted /data. You think you persisted data, but you didn’t.

Pattern B: Mounting the wrong path (typo, wrong image expectation)

You mount /var/lib/postgres but the image uses /var/lib/postgresql/data. The database writes elsewhere.

Pattern C: Mounting over the directory that contained important seeded data

You mount a volume at /app and hide the application files that were baked into the image. The container “works” but behaves differently, or fails.

Pattern D: Anonymous volume created unintentionally

You used -v /var/lib/mysql (missing the source) which creates an anonymous volume. Data persists, but you can’t easily find it later, and it may be left behind on cleanup.

Pattern E: Permissions cause the app to fall back

The app cannot write to the mounted directory, so it writes to a fallback directory inside the container layer (or refuses and crashes). You later “lose” data because it was never on the volume.

3. Inspecting mounts and proving where data is going

When you suspect data is not landing on the volume you expect, don’t guess—inspect.

3.1 List containers and get the ID

docker ps

3.2 Inspect mounts from the host

docker inspect <container_name_or_id> --format '{{json .Mounts}}' | jq .

If you don’t have jq, use:

docker inspect <container> --format '{{range .Mounts}}{{println .Type .Source "->" .Destination}}{{end}}'

You’ll see entries like:

• Type: volume with Source under /var/lib/docker/volumes/.../_data
• Type: bind with Source being a host path you chose

3.3 Inspect mounts from inside the container

Inside the container, use mount or findmnt:

docker exec -it <container> sh -lc 'mount | head -n 40'

If findmnt exists:

docker exec -it <container> sh -lc 'findmnt -T /var/lib/postgresql/data || true; findmnt | head -n 40'

3.4 Prove where a specific file lives

Inside the container:

docker exec -it <container> sh -lc 'ls -l /var/lib/postgresql/data; stat -c "%n -> inode:%i dev:%D" /var/lib/postgresql/data'

If the directory is a mount point, you’ll often see a different device ID (dev:%D) compared to nearby paths.

4. Reproducing “data loss” on purpose (so you recognize it)

4.1 The “restart test” trap

Create a container, write data, restart it, and see the data still there:

docker run -d --name restart-trap ubuntu:24.04 sleep infinity
docker exec restart-trap sh -lc 'echo "I will fool you" > /root/state.txt'
docker restart restart-trap
docker exec restart-trap sh -lc 'cat /root/state.txt'

Now remove and recreate the container:

docker rm -f restart-trap
docker run -d --name restart-trap ubuntu:24.04 sleep infinity
docker exec restart-trap sh -lc 'ls -l /root/state.txt || echo "gone"'

That “gone” is the real test. If you redeploy by replacing containers, you must use volumes.

4.2 The “wrong mount path” trap

Simulate an app writing to /var/lib/app/data, but you mount /data:

docker volume create appdata
docker run -d --name wrong-path \
  -v appdata:/data \
  ubuntu:24.04 sleep infinity

docker exec wrong-path sh -lc 'mkdir -p /var/lib/app/data; echo "important" > /var/lib/app/data/file.txt'
docker exec wrong-path sh -lc 'echo "other" > /data/other.txt'

Now remove and recreate:

docker rm -f wrong-path
docker run -d --name wrong-path \
  -v appdata:/data \
  ubuntu:24.04 sleep infinity

docker exec wrong-path sh -lc 'ls -l /data; echo "----"; ls -l /var/lib/app/data || true'

You’ll see /data/other.txt persisted, but /var/lib/app/data/file.txt is gone. The volume worked—your mount path was wrong.

5. Mounting correctly: bind mounts vs named volumes

There are two main strategies:

5.1 Named volumes (recommended for most app data)

Pros:

• Docker manages the location
• easy to attach to new containers
• avoids host-path permission surprises (sometimes)

Create and use:

docker volume create pgdata
docker run -d --name pg \
  -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16

Verify:

docker inspect pg --format '{{range .Mounts}}{{println .Name .Type .Source "->" .Destination}}{{end}}'
docker volume inspect pgdata

5.2 Bind mounts (recommended when you want explicit host paths)

Pros:

• you can see files directly on the host
• integrates with host backup tools easily
• predictable location

Example:

mkdir -p $HOME/container-data/pg
docker run -d --name pg-bind \
  -e POSTGRES_PASSWORD=secret \
  -v $HOME/container-data/pg:/var/lib/postgresql/data \
  postgres:16

Verify on host:

ls -la $HOME/container-data/pg | head

Common bind-mount mistake: mounting a relative path you didn’t mean. Prefer absolute paths.

6. The “mount hides existing data” trap

When you mount something at a path, the mount covers whatever was in that directory in the image.

6.1 Demonstration

docker run --rm ubuntu:24.04 sh -lc 'mkdir -p /demo; echo "from image layer" > /demo/file.txt; cat /demo/file.txt'

Now mount an empty volume at /demo:

docker volume create demo-vol
docker run --rm -v demo-vol:/demo ubuntu:24.04 sh -lc 'ls -la /demo; cat /demo/file.txt || echo "hidden by mount"'

The file is not deleted; it’s just hidden behind the mount.

6.2 Real-world consequence

• You baked default config into /etc/myapp/config.yaml
• You mount a volume at /etc/myapp
• The config “disappears”, app starts with defaults or fails

Fix: Mount only the subdirectory that must be persistent, or pre-populate the volume intentionally.

7. Debugging with docker exec, findmnt, and stat

When a stateful container behaves oddly, run a structured investigation.

7.1 Step 1: Identify where the app thinks it writes

Check environment variables and config:

docker exec -it <container> sh -lc 'env | sort | sed -n "1,80p"'
docker exec -it <container> sh -lc 'ps auxww | head -n 20'

For common services:

• Postgres: SHOW data_directory;
• MySQL: SHOW VARIABLES LIKE "datadir";
• Redis: check dir in config

Example for Postgres:

docker exec -it pg psql -U postgres -c "SHOW data_directory;"

7.2 Step 2: Verify mount at that exact path

docker exec -it pg sh -lc 'findmnt -T /var/lib/postgresql/data || mount | grep postgresql || true'

7.3 Step 3: Create a “marker file” and locate it

Inside container:

docker exec -it pg sh -lc 'echo "marker-$(date -Iseconds)" > /var/lib/postgresql/data/MARKER; ls -l /var/lib/postgresql/data/MARKER'

Now, from the host, locate the volume path:

docker inspect pg --format '{{range .Mounts}}{{if eq .Destination "/var/lib/postgresql/data"}}{{println .Source}}{{end}}{{end}}'

If it’s a named volume, the source will look like:

/var/lib/docker/volumes/pgdata/_data

Check the marker:

sudo ls -l /var/lib/docker/volumes/pgdata/_data/MARKER
sudo cat /var/lib/docker/volumes/pgdata/_data/MARKER

If you can’t find the marker in the host path you expected, your mount is wrong.

7.4 Step 4: Confirm you’re not writing somewhere else

Search for recent files:

docker exec -it <container> sh -lc 'find / -xdev -type f -mmin -10 2>/dev/null | head -n 50'

-xdev keeps the search within the same filesystem, which helps detect “writes went to container layer instead of mounted filesystem”.

8. Anonymous volumes and why they confuse everyone

Anonymous volumes are created when you specify a container path without a name/source.

8.1 How you accidentally create one

This creates an anonymous volume mounted at /data:

docker run -d --name anon -v /data ubuntu:24.04 sleep infinity

Inspect:

docker inspect anon --format '{{range .Mounts}}{{println .Type .Name .Source "->" .Destination}}{{end}}'

You’ll see a random volume name (a long hash).

8.2 Why it looks like data loss

• You remove the container
• The anonymous volume might remain, but you don’t know its name
• You recreate the container and get a new anonymous volume
• Your old data is still on disk, but detached

List volumes:

docker volume ls

Find dangling volumes (not used by any container):

docker volume ls -qf dangling=true

Inspect one:

docker volume inspect <volume_name>

8.3 Recover data from an anonymous volume

Mount it into a temporary container and copy out:

docker run --rm -it -v <anon_volume_name>:/from -v $PWD:/to ubuntu:24.04 sh -lc 'ls -la /from; cp -a /from/. /to/recovered/'

Then you can attach the recovered directory to a new named volume or bind mount.

9. Permissions, UID/GID mismatches, and silent fallback directories

Even with correct mounts, permissions can make the app behave as if persistence is broken.

9.1 Symptom patterns

• Logs show “permission denied”, but the service still starts
• Data directory remains empty on the host
• App creates a new directory elsewhere (like /tmp, /var/tmp, or a home directory)
• Database initializes every start (“fresh install” loop)

9.2 Inspect ownership inside and outside

Inside container:

docker exec -it <container> sh -lc 'id; ls -ld /var/lib/myapp /var/lib/myapp/data; touch /var/lib/myapp/data/.permtest && echo OK'

On host (bind mount):

ls -ld $HOME/container-data/myapp
ls -l  $HOME/container-data/myapp | head

9.3 Fixing permissions for bind mounts

You need the host directory to be writable by the container’s user.

Find the UID/GID the service runs as:

docker exec -it <container> sh -lc 'ps -o user= -p 1; id'

If the container runs as UID 1001, you can set ownership on the host:

sudo chown -R 1001:1001 $HOME/container-data/myapp
sudo chmod -R u+rwX,g+rwX $HOME/container-data/myapp

9.4 SELinux/AppArmor considerations (Linux)

On SELinux-enabled hosts, bind mounts may be blocked unless labeled correctly.

If you see permission errors despite correct UNIX permissions, check:

getenforce 2>/dev/null || true

For Docker with SELinux, you may need :Z or :z on bind mounts (syntax depends on tooling). If you’re using plain docker run, it can look like:

docker run -d --name myapp \
  -v $HOME/container-data/myapp:/var/lib/myapp:Z \
  myimage:latest

If you don’t use SELinux, :Z is unnecessary.

10. Backups, migrations, and safe recovery steps

When you suspect data is “gone”, assume it might still exist somewhere. Avoid destructive cleanup until you’ve searched.

10.1 Don’t do this first

• docker system prune -a --volumes
• deleting /var/lib/docker/volumes manually
• recreating containers repeatedly (creates more anonymous volumes)

10.2 Locate all mounts for a container

docker inspect <container> --format '{{range .Mounts}}{{println .Type .Name .Source "->" .Destination}}{{end}}'

10.3 Backup a named volume (tar archive)

docker run --rm \
  -v pgdata:/from \
  -v $PWD:/backup \
  ubuntu:24.04 sh -lc 'cd /from && tar -czf /backup/pgdata-backup.tgz .'

Restore into a fresh volume:

docker volume create pgdata-restored
docker run --rm \
  -v pgdata-restored:/to \
  -v $PWD:/backup \
  ubuntu:24.04 sh -lc 'cd /to && tar -xzf /backup/pgdata-backup.tgz'

10.4 Migrate from anonymous volume to named volume

1. Identify anonymous volume name from docker inspect (even if the container is stopped).
2. Copy into a new named volume:

docker volume create mydata
docker run --rm \
  -v <anon_volume>:/from \
  -v mydata:/to \
  ubuntu:24.04 sh -lc 'cp -a /from/. /to/'

3. Recreate container using -v mydata:/desired/path.

10.5 If data was written into container layer

If the container still exists (even stopped), you can copy from it:

docker cp <container>:/var/lib/myapp/data ./recovered-data

If the container was removed, the writable layer is typically gone (unless you have snapshots or backups at the host level). This is why volumes matter.

11. Practical checklists

11.1 “Stateful container done right” checklist

• Identify the exact data directory used by the application (docs, config, runtime query).
• Mount a named volume or bind mount to that directory.
• Verify with docker inspect ... .Mounts that the destination path matches.
• Create a marker file and confirm it appears in the volume source path.
• Test persistence by removing and recreating the container, not just restarting it.
• Confirm permissions (UID/GID, SELinux if applicable).
• Document the volume name and mount destination in runbooks.

11.2 Rapid debugging checklist for “my data disappeared”

Run these in order:

1. Is the container recreated?

   docker ps -a --no-trunc | head

2. What mounts does it have?

   docker inspect <container> --format '{{range .Mounts}}{{println .Type .Name .Source "->" .Destination}}{{end}}'

3. Does the app write where you mounted?
   • Check config/env, then:

   docker exec -it <container> sh -lc 'findmnt -T <data_path> || true'

4. Is it an anonymous volume?
   • Look for a random .Name in mounts.

   docker volume ls | head
   docker volume ls -qf dangling=true | head

5. Permissions?

   docker exec -it <container> sh -lc 'id; ls -ld <data_path>; touch <data_path>/.permtest'

11.3 “Safe cleanup” checklist

Before deleting anything:

• docker inspect and record all .Mounts
• docker volume ls and docker volume inspect for suspicious volumes
• Backup volumes with a tar archive
• Only then consider docker volume prune (and avoid --volumes on system prune unless you’re sure)

Closing: the core rule that prevents most incidents

Always prove persistence by recreating the container and verifying the data still exists in the mounted storage.
If you can’t point to the exact mount and show the marker file in the volume/bind source, you don’t have persistence—you have hope.

If you want, share:

• the exact docker run (or Compose service snippet),
• the image name,
• and the path you believe contains the data,

and I can help you pinpoint the mount mistake and the fastest recovery path.