The Mirror: A Pull-Through Registry

Problem

A CI build stalled with no visible error. The cause was an anonymous Docker Hub pull hitting the rate limit and answering with HTTP 429 - which a retry wrapper swallowed, four outer attempts around a builder doing up to four of its own, so one failing pull could generate up to sixteen requests against the limit it was trying to outlast. Beneath that specific bug sat the structural one: every node and every build asked the public internet for the same bytes, repeatedly and forever.

Constraints

• The registry is also a dependency of the thing that runs it. Its own image lives on a registry it mirrors, and the node hosting it boots its workloads through it.
• Digest-pinned estate. Every deployment references images by digest, so anything that rewrites a manifest in transit breaks the pins and any signature attached to them.
• Authentication cannot flip in one step. Node registry credentials only apply after a reboot, and a runtime whose auth is wrong does not degrade gracefully - it stops pulling.

Design

A single zot instance fronts five upstream registries in on-demand pull-through mode: a miss fetches and caches, a hit serves locally. Digest preservation is enabled per upstream, paired with the media-type compatibility flag the registry refuses to start without - conversion to OCI would otherwise change every mirrored manifest digest.

Consumers point at it through Talos machine config. The mirror is the only listed endpoint, but it is not a hard dependency, because the platform falls back to the origin registry implicitly unless that behaviour is disabled. That default is what breaks the bootstrap circle: when the hosting node boots and the mirror is not yet running, the pull falls through and the registry's own image arrives from upstream. The fleet's recovery tooling deliberately does not use the mirror at all - its image is cached outside the cluster, so it stays available when the platform it repairs is not.

Authentication is a staged rollout rather than a flip. Anonymous read stays on while each node receives credentials and reboots, and a node only counts as migrated once it passes two separate checks: an authenticated pull of a canary repository that denies anonymous access (so success cannot come from being served as an anonymous reader), and a separate correlation between an original-upstream reference pulled on a cold node and the mirror's own request log (because the first check proves identity, not routing).

What reading the source changed

The first published version of this write-up was wrong in a way worth keeping on the record. I had written that with no retention policy configured, the cache keeps everything. That holds for tagged images. It does not hold for untagged manifests, which garbage collection removes once they age past the delay - and digest-only pull-through entries are stored untagged. A digest-pinned fleet pulling through this mirror therefore caches exactly the class of object its garbage collector is entitled to delete. Selective, pull-aware retention for untagged content exists upstream, but only from a release later than the one deployed here; until an upgrade, the levers are a wider delay window or a blanket policy that trades eviction for unbounded growth.

Two related behaviours came from the same exercise. Pulls by tag contact the upstream registry even when the content is already cached, so a warm cache is not equivalent to independence from the origin. And five upstreams currently share one flat namespace, which works because their repository paths happen to be largely disjoint - a probability rather than a design, with per-origin prefixes as the fix.

Security and reliability decisions

• Fallback is a recorded policy choice. Keeping it enabled buys bootstrap survivability and availability during mirror outages, at the cost of pulls bypassing the mirror unnoticed and reaching origin limits. A regulated or disconnected estate makes the opposite call deliberately, along with preseeded content and a tested recovery path.
• Credentials are separated by consumer. Node pull credentials, the build host's push credential, and the registry's own upstream credentials never share material, and the automation identity that synchronises secrets into the cluster is read-only so it cannot half-rotate a credential pair - the failure mode behind a twelve-day outage on this estate when a stored hash and its plaintext drifted apart.
• Health checks that test the path. A process returning 200 while every pull hangs is a state this deployment has actually been in. The checks that matter authenticate, request a real manifest, and confirm the request arrived through the mirror.

Outcome

Fleet-wide pulls resolve locally instead of hitting public rate limits, and the estate's disconnected posture is now honestly bounded: the mirror is a rate-limit shield and latency win today, not yet a shelf that survives a severed link. The full teardown - configuration, diagrams, failure-mode matrix, and the tests still outstanding - is published as a public case study.

Future improvements

Upgrade to a release with selective untagged retention and write the policy before the upgrade; per-origin namespace prefixes to make upstream collisions impossible rather than improbable; enable the metrics extension so cache growth and eviction are visible; and run the disconnected drill that would let the site claim WAN independence with evidence.