413 restarts, and not one could reach the fault

This is my homelab: single-node Kubernetes clusters on Talos, one operator, and a download stack whose containers share a pod with a VPN tunnel container. Nothing here describes an employer's systems. The counts and dates are from my own notes of the incidents; the general mechanism is how Kubernetes pods work, and I say where it depends on the runtime or on this workload.

A climbing restart count reads as a system trying to heal itself. Something is wrong, the platform noticed, and it keeps having another go.

Mine was having another go into an identical room. On 31 July 2026 I found the tunnel container at 413 restarts over two days and two hours. The probe was right every time it checked, the platform did exactly what it was told, and the container died and was reborn into precisely the conditions that killed it. The log said why, in the same words every time: adding ip rule 101: from all to all table 51820: file exists. A routing rule the previous instance had left behind, and the new instance could not add its own.

That rule does not live in the container. It lives in the pod's network namespace, which is created with the pod and, in the ordinary lifecycle, survives every container restart within it. The occupant changed; the room did not.

Give the probe something true to say

Two days earlier the same stack had been silently dead for six days, because the tunnel container had no probe at all. Running equalled healthy, the downloads stalled, and nothing noticed; that story is Six days of silence. So the first fix was detection, and it is still in place:

livenessProbe:
  exec:
    command:
    - /gluetun-entrypoint
    - healthcheck
  initialDelaySeconds: 60
  periodSeconds: 60
  timeoutSeconds: 15
  failureThreshold: 3

Liveness only. A readiness probe on this container would pull the web interfaces that share its pod out of their Services exactly when they are needed for debugging. Three failed checks at a minute each is a conservative verdict, and when the verdict comes the kubelet does the one thing a liveness probe can do: it restarts the container.

Which is how a probe that fixed a six-day silence produced a 413-restart loop that had been running for two days when I found it. Detection had been fixed. Remediation had not been thought about, because I assumed the restart was the remediation.

Match the primitive to the scope of the fault

Catching a failure and repairing one are separate jobs. A liveness restart replaces the container's process; it does not ask the runtime for a new sandbox, so everything the pod's containers share - the network namespace, its interfaces, its routes and its policy rules - is still there when the new instance starts. On this runtime the resolver file is shared the same way, which is its own story, told in A healthy tunnel can still break cluster DNS.

Could the container have cleared the rule itself on the way in? In principle, yes: a process in the namespace can delete a rule it finds there, and I tried exactly that, a wrapper that flushed the stale rule before the entrypoint ran. Twice, at two memory limits, it produced a different crash loop of its own, so it was backed out for good, with the reason recorded in the commit that removed it. So the claim I can make is narrower than the one I first wrote: restarting this container, unchanged, in this sandbox never removed the rule, 413 times over. The record for the night of 1 August shows what those restarts were, too - 62 of them in eleven hours, every one a clean exit after the probe's verdict, none an out-of-memory kill. The count is the observation; the cause of each termination is only as good as the exit record behind it.

Every recovery in my notes came from a new pod. That is the historical intervention, not a universal one-line repair, and the version I would run now checks its target before it acts:

kubectl config current-context
# first: the cluster you are about to act on, read back rather than assumed
kubectl -n arr get pod -l app.kubernetes.io/name=media-vpn \
  -o custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,RESTARTS:.status.containerStatuses[*].restartCount
# expect exactly one pod, owned by a ReplicaSet; zero pods, several, or no owner, and a delete is the wrong tool
kubectl -n arr delete pod -l app.kubernetes.io/name=media-vpn --wait --timeout=120s
kubectl -n arr rollout status deployment/media-vpn --timeout=180s
# the bounded wait; if it expires, the controller has not produced a replacement and you now have a different fault

The delete takes the two containers sharing the pod down with the tunnel, so the downloader's queue has to live somewhere a pod delete does not reach. Here that is a persistent volume, and the notes that record the log afterwards show the queue intact. Then verify the tunnel by what it does, not by what the pod reports: the tunnel log prints its public exit address once the handshake completes, and a Ready column proves only that the probe passed once. The recorded recoveries took about ninety seconds from delete to a new exit address. That is a measurement from a handful of incidents, not a promise.

The uncomfortable part is that no built-in probe does this. Liveness restarts containers, and there is no probe that recreates a pod. A restartPolicy change will not do it either, because the policy governs containers and the namespace belongs to the sandbox. So either the workload repairs the state itself, as it starts or as it dies, or something outside it has to notice and act. The second option, a person paged, is where the lab sat in August.

Make the alert carry the instruction

If a person is the remediation, the alert has to reach the person and say what to do. The rule that fired is a windowed restart increase: how much the restart counter grew over the last thirty minutes. It is not a count, so it clears itself once a replaced pod stops restarting, and it is not a per-second rate either; the function extrapolates across the window, which is why the threshold is a small integer and not a fraction.

# as it ran in July and August; the live rule also pins a namespace, which matters below
increase(kube_pod_container_status_restarts_total{container="gluetun"}[30m]) >= 3

It held for five minutes before paging, meaning the condition had to stay true across evaluations for that long, and the page text ended with the command to run. On 1 August it paged the phone, unprompted, on a real wedge. On 13 August I found the stack down again, thirty-five hours in, and the alert was in the firing state in Prometheus when I looked. That is evidence for one layer. Each layer is its own: the rule evaluating true, the router handing it on, the phone receiving it, a person acknowledging it, a person acting. A screenshot of the first says nothing about the other four, and I do not have a record of the phone side for every hour of those thirty-five.

What the August record does show is that the alert did its job and the outage ran until a human read it. More rules would not have shortened it. What would is something that acts on the page: a person who answers it, an escalation that keeps asking until somebody does, or an automated pod-level repair, which is the option I have deliberately not built, and will not until its retry budget and its stop conditions are written down.

One lesson arrived later. A selector in an alert rule ties the rule to where the workload lived when it was written. When the stack moved to another cluster and another namespace at the end of August, nothing in the migration asked whether the alerts still matched anything, and for a while they did not. A migration needs an alert-selector regression test the same way it needs a smoke test for the service: query the metric with the rule's own selector and demand a non-empty answer, before the old namespace is gone.

What stretched the gaps between wedges was finding a trigger. The tunnel's own health check was starving behind an uncapped downloader on a line a house move had cut to a tenth of its speed, so the tunnel restarted itself internally, tried to re-add a rule its own previous instance still held, and wedged. On 3 August I capped the downloader at about eighty-five per cent of the line and cut its connections. The three wedges before that had come in three days; the next came nine days later, with the cap in place and the config untouched, so saturation was a trigger rather than the trigger, and the stale rule remained the failure to recover. Nine days after one change, across a handful of incidents, is a correlation I act on, not a reliability figure.

Remediate where the state lives

A self-healing mechanism is worth the match between its scope and the fault's. Detect at the level you can observe, and repair at the level where the broken state lives, because a perfect detector wired to the wrong primitive is an alarm with a restart loop attached. It will run all night, and in the morning the obstacle is still on the floor.

From the lab: four wedges after the probe landed on 29 July, the last found on 13 August 2026, and every recorded recovery a new pod. The probe above is the current manifest; the alert proved itself to a phone on 1 August 2026; automatic pod-level recovery is a design I have chosen not to build yet.