A health check cannot prove a reboot happened

This is my homelab: single-node Kubernetes clusters on Talos, one operator, and nothing here describes an employer's systems. The nodes offer no shell at all; everything below runs over the node's own API.

Most automation confirms a reboot by asking the machine whether it is awake. Issue the restart, poll the API, take the 200, mark the step green. It reads as rigour. You did not fire and forget; you checked.

You checked the wrong thing. Knock on a gatehouse at shift change and someone answers, but it may be the guard going home, still packing his bag. An answered door proves the post is manned. It says nothing about the handover.

That is the gap talosctl apply-config --mode=reboot opened in my lab. The command returns immediately, and the node's API can answer the first poll before the reboot cycles at all. Same address, same certificate, same cheerful response, served by the boot you were trying to end. Stop asking whether anyone is there. Ask which shift answered.

Ask which boot answered

"Is the API up?" and "did the machine reboot?" are different questions, and an availability check only ever answers the first. The failure mode is not an error. It is a success message about the wrong world.

The honest probe compares identities. Linux gives every boot a random ID in /proc/sys/kernel/random/boot_id, and Talos will hand that file back over its own API. Record it before the change, then wait, against a deadline, for a different one:

set -euo pipefail
NODE=10.0.0.10                    # lab example address
CFG=clusterconfig/lab-node.yaml   # lab example config
DEADLINE=${DEADLINE:-600}         # seconds to wait for a new boot
UUID='^[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}$'
boot_id() { timeout 15 talosctl -e "$NODE" -n "$NODE" read /proc/sys/kernel/random/boot_id; }

if ! BEFORE=$(boot_id) || [[ ! $BEFORE =~ $UUID ]]; then
  echo "no readable boot id before the change" >&2; exit 1
fi
timeout 60 talosctl -e "$NODE" -n "$NODE" apply-config --mode=reboot -f "$CFG"
end=$((SECONDS + DEADLINE))
while ((SECONDS < end)); do
  NOW=$(boot_id 2>/dev/null) || NOW=
  if [[ $NOW =~ $UUID && $NOW != "$BEFORE" ]]; then echo "new boot: $NOW"; exit 0; fi
  sleep 10
done
echo "no new boot within ${DEADLINE}s, still $BEFORE" >&2; exit 1
# exit 0 only on a different boot ID; a failed apply, an unreadable ID or the deadline all exit 1

My first version of this gate was an uptime threshold: under 400 seconds meant rebooted. It is wrong in both directions. A node that restarted for some other reason a minute before the change passes without ever cycling, and a slow recovery read after the threshold fails a reboot that did happen. A constant cannot tell you which boot you are looking at. An identity can.

A new boot ID is necessary, not sufficient. It says the machine cycled. Whether it came back Ready, and whether it is running the config you applied, are the next two checks, and each needs its own evidence.

What this block has been through: stub fixtures for an unchanged ID, a change after a slow recovery, a failed apply, no API, a hung call, malformed and multi-line replies, and the deadline, plus a read of the boot ID over the Talos API on the lab's Talos 1.13 nodes on 30 September 2026. It has not yet been run through a real reboot.

Learn the shape of a false green

Once you have seen one, they turn up everywhere: checks that measure availability when the question was transition.

A reconciler with self-heal on will revert an out-of-band change on its next pass, so a smoke test fired straight after the change can pass against a state that is about to stop existing. A pull-through registry shows the same gap one layer down:

MIRROR=https://registry.example.org   # lab example address
curl -sS --max-time 10 -o /dev/null -w 'api: %{http_code}\n' "$MIRROR/v2/"
curl -sS --fail --max-time 10 -o /dev/null -w 'manifest: %{http_code}\n' \
  -H 'Accept: application/vnd.oci.image.index.v1+json' \
  "$MIRROR/v2/library/busybox/manifests/latest"
# api: 200 or 401 both mean a registry answered; manifest: 200 means it served that one manifest

A registry answers /v2/ with 200 or 401 depending on whether it wants credentials, so the first line only says something is listening. The second asks for a manifest (anonymously, so an authenticated mirror needs its credential added), and a mirror whose upstream has gone away can fail it while the first stays green. Even a 200 there is a manifest check, not a pull: a mirror can hold a cached manifest while a layer behind it is missing. The full proof is the real client pulling a small, agreed image by digest into an empty store.

In every case the weak check is satisfied by the old state as well as the new one, and "never changed and still healthy" is the dangerous result, because nothing looks wrong.

Name the artefact before you automate the step

The fix generalises. Whenever a step's job is a transition (reboot, deploy, failover, rotation) the gate has to observe something only the new state can produce: a new boot ID, the new build's content hash as served from the edge, the new certificate's serial on the wire. Identity of the result, not liveness of the service.

If you cannot name the artefact that distinguishes after from before, you have not defined the step well enough to automate it.

The principle

Verify the transition, not the symptom. Write down what the old state looked like, wait for proof that something else replaced it, and treat running out of time as the failure it is.

Anyone can answer a knock. Ask which shift is on the door.

From the lab: during the June and July 2026 CNI migration a node answered its first poll on the old boot, which is why a gate stood between each reboot and the install that followed. That gate was the uptime threshold. The boot-ID version above replaced it on 30 September 2026, after review.