#Tag
Posts tagged "monitoring"
A metric can be accurate and still be history
After a maintenance window my not-ready panel went red over a cluster where nothing was wrong. It was counting Succeeded pods still held in the API. The fix is an exclusion filtered by value for readiness and a separate freshness check for backups, and both expressions now have unit tests.
The alert that could never fire
A readiness alert that passed review was watching for a series the remote cluster's collector filters out before it ever leaves. An alert you never test is a belief, not a control.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.