#Tag
Posts tagged "alerting"
A metric can be accurate and still be history
After a maintenance window my not-ready panel went red over a cluster where nothing was wrong. It was counting Succeeded pods still held in the API. The fix is an exclusion filtered by value for readiness and a separate freshness check for backups, and both expressions now have unit tests.
Two clusters, one view, and the label that decides whether it works
Forwarding metrics from every cluster to one Prometheus does not give you a fleet view; it gives you one Prometheus full of series that cannot say where they came from. The whole design turns on a single label, and the part everyone misses is that the hub's own scrapes never pass through the forwarding config that stamps it.
The alert that could never fire
A readiness alert that passed review was watching for a series the remote cluster's collector filters out before it ever leaves. An alert you never test is a belief, not a control.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.