Liveness probes kill things that would have lived

This is my homelab: single-node Kubernetes clusters on Talos, one operator, and nothing here describes an employer's systems. The tunnel in this story is the outbound-only Cloudflare Tunnel that carries this site to you.

Liveness probes are sold as self-healing. The kubelet challenges the process, the process stops answering, a restart brings it back.

The premise hiding in that sentence is the uncomfortable one: that a process which misses three challenges is dead. A sentry with those orders cannot tell a casualty from someone getting back to their feet.

On 2026-07-20 that premise cost me the tunnel. cloudflared, the tunnel client, holds outbound connections to the provider's edge and re-establishes them by itself when they drop. The edge blipped, the client's /ready endpoint started failing, and liveness was set to give up after about ninety seconds of that: a failure threshold of three and a one-second timeout, both Kubernetes defaults I had never overridden, against a thirty-second period I had set myself. Both replicas were restarted together, exit code 137, reason Error.

A sentry has to outwait the man getting up. Mine gave him ninety seconds.

Diagram in two parts. Before the fix: an edge blip left the tunnel client's /ready check failing, liveness gave up after three misses at thirty seconds with a one-second timeout, roughly ninety seconds, and both replicas were restarted with exit code 137 and reason Error. After the fix, timings approximate: readiness every ten seconds only marks the pod NotReady, because no Service routes to the tunnel client; the startup probe still allows about a minute, a cold-start path not tested; once started, liveness allows six misses at thirty seconds with a five-second timeout, roughly one hundred and eighty seconds, before restarting a process that is still failing

Treat the exit reason as a lead, not a verdict

137 is 128 plus 9: the process ended on SIGKILL. The container's last state narrows who sent it, and the events say more while they last:

kubectl -n cloudflared get pods -o jsonpath='{range .items[*]}{.metadata.name}{range .status.containerStatuses[*]}{" "}{.name}:{.lastState.terminated.reason}:{.lastState.terminated.exitCode}:{.lastState.terminated.finishedAt}{end}{"\n"}{end}'
kubectl -n cloudflared get events --field-selector reason=Killing
# line one: each pod, then each container with how and when it last ended
# line two: the kubelet's own record of a probe-driven kill, if the event has not expired yet

OOMKilled beside 137 is the runtime's own attribution to the memory cgroup, and it sends you to limits and heap profiles. Error beside 137 says less. The process ended on SIGKILL for some other reason: a liveness restart whose grace period ran out, a node going down, a runtime kill, something inside the container. It is a fork in the investigation, not an answer.

The events are where the probe names itself: Unhealthy for each failed check, Killing with a message that says the container failed its liveness probe. By default the API server keeps events for about an hour. My July notes record the exit code, the reason and the restarts piling up, not those events. What ties this incident to liveness is the pattern: both replicas restarting together, after a blip longer than the budget the probe allowed. That is a strong reading, not a proof.

Rule out a node reboot before blaming anything in the pod, and compare boot IDs rather than asking the API when you do.

Let the self-healer finish

The restart had nothing to offer. /ready on the tunnel client reports whether at least one edge connection is up, so a genuine failure means every connection is down, and at that moment the client is already working on exactly that problem. A restart gives it no new route to an edge that is not answering. It pays for a process start, throws away whatever state the process held, and then asks the same edge the same question. It does not reconnect any faster; it churns the tunnel.

None of this is specific to tunnels. If a component's core competence is recovering from the exact condition your probe measures, then that probe is measuring recovery and calling it death. An endpoint that reports connection health rather than process health can still drive a restart, but only on a budget longer than the recovery it is watching.

Give the restart the longest budget

The pod already had three probes, all asking /ready the same question. What was wrong was the tempo: the one probe that restarts a running container had its threshold and timeout left on defaults.

This is the probe block as it runs now. The fix was the last two lines:

# once started, liveness needs about 6 misses x 30s before it restarts;
# the startup probe itself still allows only about a minute
startupProbe:
  httpGet:
    path: /ready
    port: 2000
  failureThreshold: 30
  periodSeconds: 2
readinessProbe:
  httpGet:
    path: /ready
    port: 2000
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /ready
    port: 2000
  initialDelaySeconds: 15
  periodSeconds: 30
  failureThreshold: 6
  timeoutSeconds: 5

Same endpoint, three tempos. Readiness in this deployment routes nothing. No Service sits in front of the tunnel client, and which of its outbound connections carries a request is the provider's decision, not the kubelet's. A failing readiness check marks the pod NotReady, which shows in kubectl get pods and holds up a rollout, and that is all. Where a Service does select the pods, readiness is what takes a replica out of its endpoints and puts it back.

For a container that has already started, an edge outage now has to last roughly three minutes (six periods, give or take probe timing and the termination grace period) before liveness restarts it, and a process that is genuinely hung still gets restarted, which is the job liveness exists to do. The timeout moving from one second to five stops a briefly busy endpoint from counting as a failure.

Two limits, because probe tuning invites cargo-culting. First, this only covers containers that have already started. The startup probe asks the same /ready question with about a minute's budget, so a pod that starts in the middle of a long outage, after a node reboot say, can fail startup and be restarted before liveness is even in play. I have not tested that path. Second, the numbers are the shape, not the constants. The right liveness budget is longer than your component's honest worst-case recovery and shorter than your tolerance for a wedged process. I have not measured the tunnel's worst case. 180 seconds is a judgement, and the next long blip is the test of it.

The principle

A liveness probe exists to catch a deadlock. Pointed at a condition the process recovers from by itself, it catches the recovery instead, and the restart costs whatever work was in flight.

Before accepting a liveness probe's defaults, ask the question the kubelet never will: what would have happened if we had waited? If the honest answer is "it would have come back", give it the time.

Fixed in the lab on 2026-07-22, two days after the blip that restarted both replicas. The probe block above was read back from the running Deployment on 2026-09-28 and matches the manifest in git; that proves the configuration, not how it behaves in the next outage.