#Tag
Posts tagged "operations"
The server took the new limit. The node kept the old one.
I raised a transcode node's worker limit through the server's database endpoint, the server reported the new value, and the node kept behaving as before until I restarted it. That gave me a recovery step, not the reason the update never reached the running node.
If your automation opens an SSH session, it isn't automation
A play that opens a shell and greps the output is screen-scraping with a schedule attached, and it reports changed on every run because it has no idea whether it changed anything. The same work rebuilt against a typed API buys idempotency, a real preview and a change count you can believe.
Every build succeeded. The dashboard said they all failed.
A CI runner executed flawlessly for a month - images pushed, deploy branches landed - while reporting failure to the dashboard every time. A three-post release sat stranded for a fortnight because the status was believed over the artefacts.
Powered off is a state, not an incident
Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night, and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear.
The bastion is a pattern, not a box
Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that, not by being special hardware, but by being the one place the keys live and the discipline that they never leave it.