Six days of nothing, and nothing noticed

It was found by accident. Someone asked how the downloads were going, and the answer was that they had not gone anywhere in six days. The tunnel that carries them had failed and the process responsible had kept running - alive, responsive, and completely useless.

Nothing alerted, because nothing was watching the tunnel. The container was up. The platform had no reason to think otherwise.

Running was never the job

The gap between "the process is running" and "the process is doing its job" is where invisible outages live. A tunnel client with a dead tunnel is still a healthy process. It answers, it holds its port, it logs. It simply is not carrying anything, and no default check in the stack has an opinion about that.

Worse, the failure was self-obscuring. Its internal retry backoff kept growing, so the logs got quieter over time rather than louder. By day six, the system was producing almost no evidence that anything was wrong.

Health lives in the flow

What closed the gap was probing the capability rather than the container: the tunnel's own health check, which only passes when traffic genuinely egresses through it. That converts a silent failure into a restart and, eventually, an alert - a thing the platform can see and act on.

The general rule: for anything whose value is a flow, health means the flow works. Ask whether data recently moved, not whether the mover is alive.

The question that found it

Health checks should assert the outcome the component exists to produce. A running process proves almost nothing about a system whose job is to move something, and a failure nobody is watching for will always be discovered the way this one was - by a person, on the day they happen to ask.

Live in the lab: the tunnel's health check gates the container now - nothing reports healthy unless traffic egresses through it.