Stage every lock before you turn one

Every node in this fleet pulls its images through one registry: a pull-through mirror fronting five upstreams behind a single host. Anonymous read has been on since the day it went up - the bootstrap-era convenience you stop noticing, right up until you write the words production posture and realise the registry the entire estate boots from will hand an image to anyone who asks. The fix is not exotic: a named pull identity for the fleet, anonymous off.

And the reflex for shipping a fix like that is the reflex that works for everything else. Make the change, watch it, roll it back if it misbehaves. Flip the policy, delete a pod, watch the pod re-pull. Green means done.

Except this change doesn't fail when you make it. It fails at the next boot, and a node that can't pull at boot may not come back to tell you about it. No test run after the flip proves the fleet safe, because the only test that counts is a reboot, and the reboots that count are the ones you didn't plan. So the change gets split in time, the way a locksmith re-keys an occupied building: fit every cylinder and cut every key while the doors are propped open, try each key in its own door, and only when the last key turns do the props come out.

The fallback you'd bet on doesn't hold

On paper the naive flip looks survivable. Every node's registry config lists the mirror first and the true upstream second, so if the mirror answered a 401 you'd expect the runtime to shrug and pull direct. That expectation is exactly what the red-team refused to stake a fleet on: containerd's fallback behaviour on an auth failure is unreliable in practice - known, numbered upstream issues (#7321, #9997), not a hypothesis. And the trap has good manners. The flip goes green on the day, because every running pod already holds its images. The failure waits, patient, for the next power blip - then turns it into an outage.

So the red-team moved the flip to a window where every reboot is free: a fleet power-on, nodes coming up anyway, someone watching, a stumbling node getting attention instead of becoming a mystery at two in the morning. That's the first half of the split - this class of change never gets to surface its failure on a reboot you didn't choose.

Cut every key while the doors are propped

The second half is staging, and staging happens well before enforcement. The credential exists now. It lives in an encrypted environment file beside each cluster's config and is substituted into the node's machine configuration when the config renders - the committed file carries a placeholder, never the value. That auth block went into all six node repos in one reviewed pass, while anonymous read stayed on. Inert. A credential the registry never challenges you for does nothing at all; it sits in git, costing nothing, waiting.

Two costs are worth naming while it waits. Committing is free but loading isn't: on this platform, mirror endpoints reload live while registry auth only takes effect at a reboot, which is precisely why the flip has to ride reboots that were already happening. And the rotation bill went into the decision record at decision time: this is one shared pull identity across six nodes, so rotating it later means six config edits and a fleet of reboots. Accepted, consciously, in writing, not discovered mid-incident two years from now.

Try each key in its own door

Then each node proves itself, one at a time. The gate is not "it booted". Booting proves nothing - the doors are still propped, so anonymous pulls succeed regardless, and cached images would mask a broken credential anyway. The gate is a real pull, driven through the node's own container runtime via the management API, for an image that exists only on the mirror and nowhere upstream. A pass means one thing: this node presented the new credential and the registry honoured it. No fallback can fake that result, and no cache can serve it.

That's the discipline in a sentence: readiness is never inferred from the node coming back. Readiness is the key turning in the door, observed, per node.

The props stay in until the last key turns

Where it stands, honestly: three of the six nodes are staged and gated as of this week, all in a single window, and cheaply, because each node's credential reboot rode a reboot that another migration was paying for anyway. Changes that share a reboot cost less than changes that each demand their own; batching them into one window is among the cheapest wins fleet operations offers.

The flip itself is still blocked. Deliberately. One node is parked awaiting a hardware repair, so the door stays propped for everyone until that node passes its gate too. That isn't the plan slipping - it is the plan. The asymmetry does all the work: a staged lock you haven't turned costs nothing, while a turned lock with one unproven key costs a boot loop on the worst morning of the quarter. With arithmetic that lopsided, waiting isn't caution. It's reading the numbers.

Failures that only surface at the next boot

Registry auth is one instance of a wider class: changes whose failure surfaces only at the next boot, restart, or failover - where the machine that would report the failure is the machine the failure takes down. For that class, make-it-and-watch is not a strategy, because there is nothing to watch until rollback is no longer on the table. The split is the strategy. Stage the change inert, in config, where it can be reviewed and reverted for free. Prove it per node, under conditions where failure is cheap and attended. Enforce only on full coverage - never on most of it.

Stage the locks first; turn them last.

Live on a six-node Talos fleet: a zot pull-through mirror fronting five upstreams at one host, the fleet pull credential staged by sops-encrypted substitution into all six cluster repos, three nodes rebooted and gated with a mirror-only image pull, and anonymous read still on, deliberately, until the sixth key turns.