Docker container names are a mutex

NukeLab gives every user their own container to run simulations in. The runtime image is small — nginx plus a Python runtime — and the physics stack lives in a separate toolchain image: OpenMC, the nuclear data libraries, the cross-section sets. Several gigabytes of files that change rarely and cost a lot to pull.

I did not want each spawn to copy that stack into the server’s workspace. The node would store the same gigabytes once per server and pay a full copy every time. So the toolchain lives in a named volume, shared by every server using the same image and mounted read-only, by default at /opt/nuke.

The volume is the easy part. The cache around it is not, because it has to survive three things: an image tag that gets re-pushed, two processes spawning at once, and a copy that dies halfway.

A cache needs a key, a validity check, and a lock

Miss the key and nothing is shared. Miss the validity check and the cache serves stale data forever. Miss the lock and two writers produce a half-copied volume that looks fine until someone imports a module.

Each of those is a separate mechanism in the code, and each one has a failure mode that is quiet.

I keyed the volume on the image, plus a hash

nukelab-toolchain-<sanitised image>-<sha256(image)[:12]>

Deriving the name from the image reference is what makes it shared: every server configured with the same toolchain mounts the same volume.

The digest suffix exists for the case nobody hits until they do. Docker has a hard limit on name length, and the name is built from a sanitised image reference that can be long. Truncating at 240 characters means two different references sharing a long prefix would collapse into one name — and then one user’s container mounts a different toolchain than its template asked for. Twelve hex characters of SHA-256 over the full reference makes that collision cost 2^48 work instead of one unlucky namespace.

A tag is not an identity, so the volume carries a stamp

The volume is named after the image, but the image is named by a tag, and tags are mutable. Re-push the same tag with new content, and every existing volume still claims to hold that image. A cache keyed on the tag alone serves the old toolchain forever, and the bug presents as “the new feature isn’t in the container” — which nobody debugs as a caching problem.

So population writes a stamp into the volume root:

{
  "image_id": "sha256:…",
  "repo_digests": ["nukelab/toolchain@sha256:…"],
  "populated_at": "2026-11-11T09:14:02+00:00"
}

On every spawn, the fast path reads the stamp and compares image_id with the ID of the local image. Match, and the volume is returned without copying anything. Mismatch, and the volume is wiped and repopulated.

Wiped, not overlaid. cp -a over an existing tree leaves files that were removed from the new image sitting in the volume, which is the long way around to a mystery import error. The copy script starts with rm -rf on the target’s contents for exactly this reason.

There is one cross-engine detail worth pinning down: Docker reports image IDs as sha256:<hex>, while Podman’s CLI may report the bare <hex>. A plain string comparison makes the cache miss on every single spawn under Podman — and a cache that never hits is worse than no cache, because it still looks like it is working in the logs. The comparison strips the algorithm prefix and compares digests.

I could not put the lock in the process

The spawn path is reached from two places: FastAPI request workers and Celery tasks. They are different processes. An asyncio.Lock is per-process and therefore useless. A file lock would depend on both processes sharing a filesystem view, which is exactly what you stop being able to assume once containers are involved.

The one thing they all definitely share is the Docker engine. So the lock lives there, and it is a container name:

lock_name = f"nukelab-toolchain-lock-{sha256(volume_name)[:12]}"
await client.containers.create(
    {"Image": image, "Cmd": ["true"], "Labels": {...}, "HostConfig": {"AutoRemove": False}},
    name=lock_name,
)

Creating a container with a name that already exists fails with HTTP 409. Name creation is atomic at the engine level, so the second caller loses the race and knows it. The container is never started, so Cmd: ["true"] never runs, and AutoRemove is off because removing the container would release the lock. The name is the whole mechanism. Nothing runs.

The loser polls every two seconds and gives up after fifteen minutes with an error that names both the lock and the volume, because the person reading that error needs to know which of two things to delete.

The stale lock is the hard part

If the process populating a volume is killed — an OOM, a deploy, a docker system prune racing it — the lock container stays behind and every future spawn waits fifteen minutes and then fails.

There is no clean signal for “the holder is dead” here. The container was never started, so it has no exit code and no status to inspect: a live holder and a dead holder are identical objects. Age is the only available signal, which is why the stale threshold and the wait timeout are the same number. Anything older than fifteen minutes is assumed dead and force-deleted.

That number is a trade-off I had to choose, not a constant. Set it below the slowest legitimate populate and two processes will eventually write the same volume at the same time — the failure the lock exists to prevent, now caused by the lock. Fifteen minutes is comfortably above a multi-gigabyte copy on a slow disk and comfortably below the point where a user gives up on a spawn.

Winning the lock does not mean doing the work

After acquiring, I re-read the stamp before copying anything. While the loser was waiting, the winner finished — the volume is already populated and valid. Populating again would be correct and pointless.

Without the re-check the lock still works, and it still serializes N copies of the same gigabyte. The lock’s job is to prevent concurrent writes; the re-check’s job is to prevent redundant ones.

I made a broken cache degrade the feature, not the platform

If pulling the toolchain image fails, the spawn logs a warning and continues without it. If preparing the volume fails, the spawn continues without it and logs:

TOOLCHAIN_DEGRADED: failed to prepare toolchain image …; server … spawns WITHOUT the toolchain

A user gets a working container that can run Python but not OpenMC, plus a log line that says exactly why. The alternative — refusing to spawn — turns “the cache is broken” into “the platform is down”, which is a much worse trade for a feature whose entire purpose is performance.

The same reasoning covers the read-only mount: a container should not be able to damage the thing every other container on the node is sharing, whether or not its own view of that thing is complete.

The copy container is the one that gets to keep capabilities

Population runs in a one-shot container: rm -rf the target, cp -a each mount path’s contents in, write the stamp. Otherwise hardened — all capabilities dropped, no-new-privileges, read-only root filesystem, a 64 MB tmpfs at /tmp.

Except that it needs five capabilities back: CHOWN, FOWNER, DAC_OVERRIDE, SETUID, SETGID. cp -a preserves ownership, and preserving ownership means creating files as other users. So this container runs as the image’s default user rather than a fixed one. That is the price of an archive-faithful copy, and I wrote it down, because “drop all capabilities” is the kind of rule that gets applied uniformly and then quietly weakened later by someone who cannot work out why the copy fails.

If the copy exits non-zero, the exception carries the container’s logs. A failed background copy that only writes to a log file is a support ticket with no attached evidence.

What this is not

It is not a distributed lock. Container-name uniqueness is guaranteed by one engine, so the guarantee holds on one node. More than one node would need something else. That is a design boundary, not an oversight.

The rule I would apply to the next cache

A cache is only a cache if it has all three parts: a key that determines sharing, a validity check that detects change, and a lock that serializes the miss. Skip the key and it is not shared; skip validity and it lies; skip the lock and it corrupts. And the failure path has to be a degraded feature, never an outage — because the whole point of the cache was to make the system faster, not to become the reason it stops.