#Tag
Posts tagged "reliability"
413 restarts, and not one could reach the fault
A liveness probe restarted a container over and over without once repairing it, because the broken state lived in the pod's network namespace rather than inside the container. Detection and remediation are separate jobs, and a restart only heals when the fault sits inside the thing being restarted.
One way in, and a dead route looks like a dead estate
A broken route into a network is indistinguishable from every service in it being down, because the evidence available from outside is identical in both cases. The fix is an order of diagnosis - prove the service from a chair that is not yours, then walk the path - and a gate that is never a tenant of the building it lets you into.
Liveness probes kill things that would have lived
An edge blip set my tunnel client's health endpoint failing, and both replicas were restarted once it outlasted the budget a liveness probe on its defaults allowed, while the process was reconnecting on its own. A liveness probe should catch a deadlock, not interrupt a recovery, so its budget has to outlast the recovery it watches.
A health check cannot prove a reboot happened
Automation that confirms a restart by polling for a healthy API is asking a question the old boot answers just as cheerfully as the new one. Gate a transition on evidence only the new state can produce: for a reboot, a boot ID that differs from the one you recorded before the change.
A healthy tunnel can still break cluster DNS
Twice a shared pod in my lab has failed on a name it could not resolve, and twice the component raising the alarm was working correctly. Both times the resolver every container in that pod read had been rewritten by the tunnel container beside them, and the fastest route to the cause was testing the dependency instead of the component reporting the fault.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.
Init-gating GPU readiness on Kubernetes
The single highest-leverage reliability fix for edge GPU workloads: never let an inference pod schedule before the GPU is actually ready.