A mirror saves the download, not the dependency

This is my homelab: single-node Kubernetes clusters on Talos, one operator, one mirror registry in front of the public ones. Nothing here describes an employer's systems. Where I say a thing broke, it broke here, on a date I give; where I describe what a tool does by default, that is the tool's documented behaviour, checked against my own config rather than measured under load.

Count what happens when a pod lands on a fresh node. The container runtime checks its local store, finds nothing, and reaches across the internet for layers the cluster next door already holds. Every rebuild on a new node, every scale-up, is another download and another bet that the registry on the far end is up, unthrottled and in a good mood.

That is the default nobody argues with: a cache per node and nothing shared between them. A mirror on the LAN changes the arithmetic. A layer crosses the border once and is served from your side after that, and it is easy to hear that as "pull from the internet once". It is not. A tag still asks upstream whether it has moved. A miss still crosses. The fallback to the origin, which you want, keeps the upstream in your availability story. What the mirror really does is cheaper and more interesting: it decides what bytes the fleet is handed, and by default it changes them.

Put one cache on the border, and know its three paths

The cache here is zot, a small OCI registry with a pull-through sync extension. The public upstreams sit behind it and every node's container runtime is pointed at the mirror rather than at the world. There are three paths through it, and they do not cost the same. A request by digest for content the mirror already holds is answered off the LAN; that is the warm path, and the one the word "mirror" makes you picture. A request by tag is a freshness question first: the mirror asks the upstream whether the tag has moved before it answers, and its own log says trying to get updated image by syncing on demand for a tag it already held, which I read on 7 October 2026. A request for anything it has never seen walks out to the upstream, lands in the cache, and only then comes back. Every upstream is configured on demand only, so nothing is pre-fetched, and a cold miss is a cold miss. The original "build hang" that started all this was a CI build re-resolving an unpinned base image from Docker Hub on every run; pinning the base by digest is what fixed that, and the mirror is what made the fix cheap.

One deliberate omission keeps the arrangement safe. The runtime's fallback to the origin stays enabled, so the day the mirror itself is the thing being upgraded, its own image can still be fetched. A warehouse that has to be open before anyone can open the warehouse is not a warehouse. The one time I have seen that fallback work was unplanned: on 23 June 2026 the mirror crashlooped on a config mistake, described below, and the CI builds carried on against the real upstreams. Read that receipt for exactly what it is. The build engine has its own registry configuration, and that day the mirror was refusing connections. It does not prove that a node's container runtime, configured separately, would cold-start the mirror's own image through the same fallback, and I have not run that test.

Nor did the fallback cover a mirror that hung. Twice, on 7 July and 28 September 2026, an on-demand sync stalled on one upstream and every request behind it queued with image already demanded, waiting on channel. The nodes' pulls ran out their budget, logged context canceled, and backed off; they did not fall back, because what a client does next depends on the error it gets, and a socket that stays open past the pull deadline is not the error that sends this runtime to the origin. Restarting the mirror cleared it both times. The lesson is not a rule about fallback. It is that a mirror's failure modes have to be tested per client and per error: refused, hung, unauthorised, warm, cold, origin down. I hold receipts for two of those six, and both arrived by accident.

Serve the manifest you were given

By default, zot's sync converts Docker schema-2 manifests to the OCI format as it mirrors them. The layers and the config are unchanged; the manifest that lists them is rewritten into another dialect, and an image digest is the hash of the manifest. Rewrite the manifest and you have minted a different digest for the same bytes. One analogy, and then the plain terms: customs opened every crate, repacked the contents into its own standard boxes, and binned the original seals.

Downstream, that is the kind of failure that surfaces later and looks unrelated. A @sha256: pin asks the mirror for a digest it does not hold. A signature was made over the original manifest and meets a stranger. Nothing fails at sync time. I have not had a pin break on me, and the reason is unflattering: when I set this up the fleet carried no digest pins against the converted upstream at all. That is precisely why the setting goes in before the first pin, not after the first mystery.

Two settings keep the manifest intact. This is a fragment of the server config, not the file; the storage, log and auth sections live alongside it:

{
  "http": { "compat": ["docker2s2"] },
  "extensions": {
    "sync": {
      "registries": [
        {
          "urls": ["https://ghcr.io"],
          "onDemand": true,
          "tlsVerify": true,
          "preserveDigest": true,
          "content": [{ "prefix": "**" }]
        }
      ]
    }
  }
}

preserveDigest stops the rewrite; http.compat lets the registry store and serve schema-2 manifests as they arrived instead of insisting on OCI. They are not independent, and that is the trap. On zot 2.1.17, asking for digest preservation without the compatibility flag is rejected at startup with can not use PreserveDigest option without enabling http.Compat, and the pod crashloops. I shipped exactly that on 23 June 2026 and caught it after the merge. Since then the file goes through the server's own check before it merges, by hand rather than as a pipeline step, and I have not replayed the broken June file through it, so it is the check I trust rather than one I have watched catch this mistake:

zot verify /etc/zot/config.json
# run from the same image tag the cluster runs; a non-zero exit blocks the merge and means diagnose, not "the server would refuse it"
# a missing file or a bad invocation fails the same way, and a clean exit proves the file parses and passes the server's own rules, nothing about storage, bind, credentials or upstreams at start

Preserving the digest is necessary for a pin or a signature to survive the mirror. It is not verification. A matching digest proves the mirror handed over the same manifest bytes it received, and nothing more: not who published them, not whether a signature was replicated alongside, not whether anything checked them against an identity you trust. Replicating signatures and referrers is a sync setting of its own, and verification belongs in admission, not in a cache.

Once the registry was back, I checked from the outside: a request by digest for an upstream multi-architecture index came back 200 from the mirror with the exact upstream digest. That check only discriminates on an image the upstream serves as a Docker schema-2 list. An image published as OCI in the first place has nothing to convert and passes with the setting on or off, and my notes do not say which kind I asked for. So it is a smoke test. The test that would discriminate is a HEAD for the same index against upstream and mirror, recording Content-Type and the digest header from each, on an index whose upstream type is application/vnd.docker.distribution.manifest.list.v2+json: a different type or a different digest from the mirror means the rewrite is on. I have not run it, and until I do, "preserved" is a configuration claim with one non-discriminating receipt behind it.

The door is closed, and that was a different post

An open registry on the LAN is an open door, so anonymous read had to go as well. That change has nothing to do with digests and everything to do with ordering, because the credential a node needs to pull only takes effect at the next boot. The staging of it, inert in every node's config before the door latches, is told in Stage every lock before you turn one. The two changes are separable, and I would make them separately again: the manifest setting protects the fleet from the mirror, the lock protects the mirror from the LAN.

Keep the bytes, test the failure

Every registry you pull from at runtime is a dependency you never declared. A pull-through mirror does not remove it; the tag check, the cold miss and the fallback you want all still reach for it. What the mirror changes is how often you pay, and what the fleet is handed when it asks. So the two things to get right are the two the default gets wrong or leaves untested: serve the exact manifest you were given, and know, per client and per error, what happens when the mirror is the thing that fails. Keep the bytes identical, and test the failure you expect before the one you did not.

Live in the lab: one zot mirror in front of the public registries the fleet uses, on demand only, digest preservation on every upstream entry and the compatibility flag set once on the server, with the runtime's origin fallback left in place.