A container failed to start. The log showed a line, always the same:

exec /app/bin/boot: no such file or directory

This sounds like a broken Dockerfile. The file is missing from the image, so something is wrong with the COPY instruction. You check the Dockerfile, find nothing, check the build context, find nothing.

Then you start the image with /bin/ls instead of the Entrypoint, and the answer is: stat /bin/ls: no such file or directory. The Entrypoint isn't missing. The entire filesystem is missing. The Docker build reported success, created an image with a plausible size, and that image is empty inside.

I had an explanation for that since July. It fit every detail and was still wrong, and it took me a quarter of a year to notice it.

In July the explanation fit too well

At that time the same symptom hit a freshly set up server. Docker had recently changed the default: new installations from version 29 store images in the containerd store instead of the classic one. And in Docker's bug tracker there was an open issue about exactly that store, where image export fails on large builds — moby/moby#52431, open since April.

The chain closed immediately. New server, new store, known bug. We nailed the classic driver in the daemon.json firmly, rebuilt, and the problem was gone.

What I didn't do back then: check if the explanation was correct. It fit, the problem disappeared, and that was the end of the case.

How a hypothesis becomes a fact

It is written down. First as a comment in the setup script, directly above the line that sets the old driver. The same reasoning then appeared in the project notes, in the operations documentation, and in several commit messages.

None of these places was a measurement; each was a copy of the previous one.

This is most clearly seen in a sentence from my notes: The pin must not be removed until the upstream bug is closed. That sounds like diligence. But it means that I had made the state of my own servers dependent on a foreign bug tracker.

In addition, there is my tool. I work with Claude as an AI assistant, and he reads exactly these files. When the symptom recurred in August, he suggested a server restart, soundly justified by the notes from July. Whether the notes were correct, he did not ask.

This is not a flaw of the model. It establishes consistency with what it finds. If what is found is wrong, the error is thereby only more thoroughly distributed.

In August, the explanation was no longer valid.

The server that was hit this time had been running on the old driver for a month. Precisely the one that should have prevented the problem.

Thus, the hypothesis was essentially already dead. A protection that does not prevent the problem is no protection. Nevertheless, I almost solved it again with an attempt at explanation rather than with a measurement, because the restart sounded plausible and there were corresponding messages in the kernel log.

The question that resolved the node came from a human. It amounted roughly to:Could it not be that this was a one-time slip-up and the new store has long been in order?

An afternoon of measuring against a quarter-year of guessing.

The setup was simple. A disposable instance with Docker 29.7.2, two runs, each with five rounds using an image of 2.2 gigabytes and 22 layers, built cold, warm, and after the cleanup command that our update tool executes anyway.

Two things about it were more important than the setup itself.

First, I had the decision rulesin advance.written down, including the results that would go against me. If the new store runs flawlessly, the pin is out. If both runs fail, it was never the cure. Whoever sets the evaluation rule only after seeing the numbers will always find one that fits their own thesis.

Second, there was a control arm. The new store could have also run cleanly by accident.

The result: five out of five flawless on the new store, not a single kernel message. The control arm with the old driver also five out of five. Both stores build usable images, and the hollow images come from neither.

The pin came back, for a different reason

The next day I measured the real use case instead of a test image: the full build of our application, with archives located locally beforehand, as it runs in production. Three cold and two warm rounds per run, same machine.

Cold 14 seconds versus 37. Export alone 3.5 versus 19.7. And the line that decided: the warm build, which should have taken mere fractions of a second, required on the new store the same 36 seconds as the cold one. Its build cache does not survive the cleanup command that our tool executes after every update. This is my measurement on one machine, not a documented property of the product.

So the old driver is back in the setup script. For a measured reason, with the numbers beside it, and with a switch that disables it.

The original cause of the hollow images remains unknown to this day. I have a hypothesis that fits the timestamps, and I have written it down as a hypothesis in the notes, along with the command that will confirm or refute it upon the next occurrence.

That's the only difference from July.

The measurements

Two runs on two days, both on the same disposable instance: Debian 13, Docker 29.7.2. The top half answers the question about hollow images, the bottom about speed. Only the bottom brought back the pin.

The synthetic run was on 2026-08-14 with random data that cannot be compressed, and thus consists almost entirely of the export step. The Odoo build from 2026-08-15 is the real use case: 394 module archives were pre-loaded locally, three cold and two warm rounds per run.

The last line is the one that matters. Our update tool calls docker system prune -f after each run, and the build cache of the containerd Store does not survive this — every update would be a full rebuild there.

The figures in the second-to-last and third-to-last rows are meant to be read together: If the total times had differed, without the export step moving along, the storage driver would not have been the cause. It moves along to account for the factor 5.6.

Created by Martin Schmid, with support from Claude Opus 5 and released after its own content review. Our Notes and Disclaimer.