#Tag
Posts tagged "debugging"
413 restarts, and not one could reach the fault
A liveness probe restarted a container over and over without once repairing it, because the broken state lived in the pod's network namespace rather than inside the container. Detection and remediation are separate jobs, and a restart only heals when the fault sits inside the thing being restarted.
One way in, and a dead route looks like a dead estate
A broken route into a network is indistinguishable from every service in it being down, because the evidence available from outside is identical in both cases. The fix is an order of diagnosis - prove the service from a chair that is not yours, then walk the path - and a gate that is never a tenant of the building it lets you into.
Three empty builds before I read the kernel's file
A per-engine GPU exporter for two consumer Intel Arc cards took five builds in one evening, and the three before the first real row came back empty because each trusted a reading borrowed from somewhere else. The row arrived only after the kernel's own file was read off the working card, which is where any parser for an interface you have not read yourself should start.
Liveness probes kill things that would have lived
An edge blip set my tunnel client's health endpoint failing, and both replicas were restarted once it outlasted the budget a liveness probe on its defaults allowed, while the process was reconnecting on its own. A liveness probe should catch a deadlock, not interrupt a recovery, so its budget has to outlast the recovery it watches.
The form supplied defaults my script never sent
Create an OAuth2 provider through Authentik's admin form and logins work; create it in the admin shell and every login comes back as a malformed request, because the form was ticking boxes the model leaves empty. Automation that writes underneath a form inherits the form's job, and a working object shows what that job includes without deciding it for the next app.
A healthy tunnel can still break cluster DNS
Twice a shared pod in my lab has failed on a name it could not resolve, and twice the component raising the alarm was working correctly. Both times the resolver every container in that pod read had been rewritten by the tunnel container beside them, and the fastest route to the cause was testing the dependency instead of the component reporting the fault.
fsGroup is a request, not a guarantee
A non-root pod crashlooped on permission denied with a textbook securityContext, because the volume plugin behind a host-directory-backed claim carries no ownership pass and nothing says so. The field validated, the ownership change never happened, and only the directory reached through the application's own mount could settle it.
A finished scan can have walked nothing
A library scan answered 200 and its scanner logged a launch and a finish about eighty milliseconds apart with zero files found, because one parameter was an empty array where a folder path belonged. The tool had done exactly what it was asked. Only an inventory taken outside it, set against its own records, showed two libraries 164 files short.
The server took the new limit. The node kept the old one.
I raised a transcode node's worker limit through the server's database endpoint, the server reported the new value, and the node kept behaving as before until I restarted it. That gave me a recovery step, not the reason the update never reached the running node.
Every build succeeded. The dashboard said they all failed.
A CI runner executed flawlessly for a month - images pushed, deploy branches landed - while reporting failure to the dashboard every time. A three-post release sat stranded for a fortnight because the status was believed over the artefacts.