If your automation opens an SSH session, it isn't automation

Look at how most infrastructure automation reaches its target. It opens a shell. It runs a command a human would type, captures whatever came back, and pattern-matches the text for a word that means success. The play that "checks node health" is ssh to a box, kubectl get nodes, grep Ready. The play that "reconfigures the deployment" is ssh to the same box and kubectl apply from a file it copied there a moment earlier.

That is not automation. It is a human session with the human removed, and it inherits every weakness of the session it imitates. The output format is a courtesy, not a contract, and any tooling update can change it. There is no state to compare against, so the play cannot tell you whether it changed anything. There is no preview mode, so the first run is the real run. And a shell that can run kubectl apply can run everything else, which means the credential your scheduler holds is a root-shaped hole in the wall.

So stop reaching through the letterbox. Every system worth automating has an API on the other side of the door, and the API was designed to be turned like a key.

Turn the key instead of reaching through the slot

Rebuilding those plays against the real API changes the shape of the work, not just the transport. You stop parsing what a command printed and start declaring what should be true, then let the far side merge the difference:

- name: Ensure the restore-drill job matches git
  kubernetes.core.k8s:
    namespace: backup
    state: present
    apply: true
    server_side_apply:
      field_manager: fleet-ansible
    src: manifests/restore-drill-job.yaml
    wait: true
    wait_condition:
      type: Complete
      status: "True"
    wait_timeout: 120
# changed only when the server merged a difference; the second run of the same play must report changed=0

That task is a no-op on every run after the first. Idempotency stopped being hand-carved logic and became a property of the interaction, and the play does not return until the job it created has finished, because wait_condition is a fact the API reports rather than a sleep somebody guessed.

Here is the one I found in the documentation before it found me. In kubernetes.core 6.5.0, the version the lab pins, server_side_apply is ignored unless apply: true sits beside it. The task still succeeds, the object still lands, and the field manager you thought you had registered is not there when a conflict finally matters. Two related habits keep the honesty: gate on a condition (type, its expected status, optionally a reason) rather than an until loop, and leave force_conflicts at its default of false unless you mean it, because it steals fields from whichever manager wrote them last.

The same rule holds beneath Kubernetes. The nodes run an immutable Linux with no SSH daemon at all; I have written before about why that is the feature. For automation the consequence is that every node action goes through the Talos API, and the health gate that fronts every play is one typed call:

- name: Talos health
  ansible.builtin.command:
    argv:
      - talosctl
      - --context
      - cave
      - health
      - --wait-timeout
      - 60s
  changed_when: false
# a read has no side effect, so it must say so; without changed_when every run reports a change it never made

Make the second run prove the first one

Speed is the win everyone expects from this rewrite. The one that matters is honesty.

Shell-based automation reports changed on every run because it has no idea whether it made a difference. A change count that is never zero carries no information; it is a number the eye slides past. Once the count means something, a surprising one becomes a reason to stop and look.

So make the preview run first and the second run the gate:

ansible-playbook playbooks/estate_health_check.yml --check --diff   # what would change, without changing it
ansible-playbook playbooks/estate_health_check.yml                  # first run
ansible-playbook playbooks/estate_health_check.yml                  # second run must report changed=0
# a non-zero count on the second run means a task is lying about its effect; the preview must run clean, not skip

A non-zero count on that second run means a task is lying about its own effect, and it will keep lying on the night it matters. The one play in the lab allowed a non-zero count on every run is the restore drill, because it creates a fresh job each time by design and is named for what it does. Everything else converges to zero. The preview on the first line is the other half of the contract. When the play declares state through the API, --check --diff is nearly free; when it drives a shell, there is no preview to give, because the tool being wrapped never had one.

One caveat, because it bit me. In check mode the tasks that gather facts are skipped, and every assert downstream of them fails on an empty variable. Giving up the preview is the wrong fix. Mark the read-only tasks check_mode: false (they read, so they are safe to run in a preview) and changed_when: false (they read, so they never changed anything). The lab's health gate does exactly that for its node, deployment and endpoint reads, and the preview promise holds for the whole play.

Draw the line between the reconciler and the verb

A cluster that already has a reconciler needs one line drawn before you write a single play against it. GitOps owns steady state: what should exist, with what config, converged continuously from git. Ansible owns the verbs that do not belong in a continuous loop: bring the fleet up in the right order, run a restore drill, audit drift, gate a change on the health of everything around it.

Blur that line and you get two writers arguing over one object. The reconciler applies what git says; a play patches it back; the reconciler notices and reverts; the play runs again on schedule. Nobody wins, and the diff churn buries every real change under the noise of the fight. The typed API makes the argument visible at least, because a server-side apply records which manager owns which field and refuses a silent overwrite. The shell version just clobbers.

The verbs also have to run from somewhere, and it cannot be the estate they exist to recover. A runner that lives on the cluster it operates has a dependency on the thing it is supposed to recover. When the estate is dark, the tool that brings it back cannot be hosted on the estate. So the runner is off-fleet: a small VM that holds nothing but the plays, the collection pins and a scoped credential, and reaches everything it touches over the same APIs everything else uses. Bringing the fleet up is a play; recovering the runner is a copy of a small repository.

The principle

Automate against interfaces designed to be automated. A wrapper around a terminal works right up until the output format moves, and no amount of careful parsing buys what a typed API gives away for free: state you can compare, changes you can preview, reports you can believe.

The letterbox was only ever cut for envelopes. Anything that needs to get in has a key.

In the lab: the health gate, the restore drill and the power-on play run from an off-fleet runner through the Kubernetes and Talos APIs, pinned to kubernetes.core 6.5.0, and a zero-changed run means what it says.