Powered off is a state, not an incident

Homelabs run 24/7 out of habit. The habit comes with a scoreboard - uptime counters climbing into the hundreds of days, worn as proof of seriousness, and with two feelings nobody says out loud: the guilt of turning a server off, as if a dark machine means you weren't serious; and the quieter fear that if you do turn it off, it won't come back. Somewhere on that box, the fear whispers, is a state nobody wrote down, held together by warmth.

Meanwhile the actual work is bursty. GPU inference runs when a request arrives. Transcode runs in batches and finishes. The hardware idles around the clock for workloads that occupy it a fraction of the time - real watts, real fan-hours, real heat, spent keeping silicon warm for nobody.

Last night I shut down half the fleet on purpose. Three machines - two of them carrying an RTX 5080 each - were upgraded in the afternoon, verified healthy in the evening, and powered off before midnight. Nothing broke, nothing was lost, and the fleet's books record the event as a state, not a failure. Think of a building at night: the porter walks the floors, the lights go off in the wings nobody's using, and the building isn't failing. It's resting, and the porter writes it in the log.

Make the node a document, and power becomes a detail

None of this is safe by default. It's safe because of a property I argued for in the sibling post: each of these machines is a single-node Talos cluster whose entire existence is declared. The machine config pins the disks by serial and the network interfaces by MAC; the CNI ships inside that same document as an inline manifest; the workloads arrive from git the moment the node reports Ready, reconciled by the fleet's GitOps controller; the credentials are substituted in at render time from an encrypted file. There is no state that exists only on the box. The node is a document - power is just whether it's currently being read.

That property is what made this week's sequence unremarkable. Three nodes had their network datapath swapped - kube-proxy out, Cilium's eBPF replacement in, and their registry credentials rotated, one reboot each. Every check came back green: GitOps fully synced, secrets flowing, every endpoint answering. And then, the same night, all three were shut down. Upgraded, then slept. Nothing about the upgrade is lost to the power state: it's on disk, in the config, in git. When the power button is next pressed, each node boots into exactly the machine its documents describe - including everything that changed yesterday.

How they went down matters as much as the fact that they could. talosctl shutdown is a clean, API-driven stop: workloads terminated properly, etcd closed cleanly, the platform powering itself off - confirmed dark before anyone walked away. A graceful shutdown is what makes the next boot boring. Pull the plug instead and you've converted a healthy state into a small forensic exercise for future-you.

Keep the two kinds of dark apart

Four of the six machines are dark tonight, and the fleet's records refuse to treat them as one category.

Three (robin, kate and wgirl) are off by choice: healthy at shutdown, upgraded that same day, one press of a power button from Ready. The fourth, elfastc, is parked broken: dark because of a hardware fault, waiting on a screwdriver, not a switch. From the network's point of view the four are identical: no link light, no metrics, nothing to distinguish rest from wreckage. From the operator's they are different states with different exits, and the roster records which is which, in the same versioned pages as everything else.

This is the porter's log. A dark wing and a broken lift look the same from the street; the difference is written down at the desk. Six months from now, "why is this node off?" must have an answer that isn't archaeology, because a fleet that can't tell resting from broken will eventually treat one as the other, and both mistakes are expensive: rebuilding a healthy node because nobody trusted it, or trusting a broken one because it merely looked asleep.

Pay the costs where you can see them

Sleep isn't free, and pretending otherwise is how the 24/7 habit wins the argument. Two costs are real, and both are stated in the books.

First: what lives on a sleeping node is dormant, not migrated. The fleet's LLM backend lives on wgirl's GPU, and it sleeps when she does. The assistant app that depends on it stays up - it runs on a node that's awake, but until power-on it has no model to call. That is a genuine loss of capability, accepted deliberately, recorded next to the state that caused it. If the trade ever stops being worth it, the fix is a power button, not a redesign.

Second: monitoring fires as a node goes silent. The metrics stream stops mid-sentence and the alert raises, exactly as it should - the monitoring has no way to know the silence was chosen, and it must never assume so. The wrong response is to call that a false alarm. It isn't. It's the system being honest about a deliberate act, and the deliberate act is written down where the alert can be checked against it. The porter doesn't disable the alarm panel for the dark wings - he checks the panel against the log.

Design for off

Design for off. A node you can power down without ceremony is a node whose whole truth lives outside it - in documents, in git, in the log, and that is precisely the property you need on the day a node goes down without asking. The fleet you can put to sleep on purpose and the fleet that shrugs off losing a node in anger are the same fleet; the two abilities are one property. Uptime measures how long a machine has been running. It says nothing about how well you hold it. A building isn't failing when its lights go out at night - it's failing when nobody dares touch the switch.

Live in the fleet's books tonight: three of six Talos nodes - robin and wgirl (an RTX 5080 each) and kate (the AI sandbox) - upgraded to the new datapath, verified green, then gracefully shut down through the API and recorded off-by-choice; elfastc parked-broken awaiting a screwdriver; cave and alfred carrying the always-on core.