The alert that could never fire

Each remote cluster in this fleet forwards its metrics through an allowlist: a deliberately short list of the series worth shipping - a lean collector and a small central store, bought at the price of everything else staying home.

The readiness metric was not on it. The rule I wrote against readiness loaded cleanly, read correctly, and passed review. It also could never fire: the series it watched for is filtered out before it ever leaves the cluster it describes.

The rule was not wrong. It was unreachable.

Valid, deployed, and permanently silent

Nothing about that fails loudly. The rule loads. Its syntax is fine. Its expression evaluates against no series and returns nothing - and returning nothing is also what health looks like. A rule that can't fire and a rule that has no reason to fire are indistinguishable from the outside.

That's the whole hazard: a monitoring gap disguised as good news, sitting in version control looking like coverage.

Query before you author

The check costs thirty seconds. Before writing a rule, query the metric against the system that will evaluate it - the central store, not the cluster that emits it - and confirm series come back:

curl -sG http://prometheus.hub.lan/api/v1/series \
  --data-urlencode 'match[]=kube_pod_status_ready{cluster="robin"}' | jq '.data | length'
# 0 = the rule you are about to write can never fire

Not "does this metric exist" in general - does it exist there, having survived every filter between its source and the evaluator.

Here that check redirected the whole design. Readiness was unavailable; restart count was present. So the rule keys on restarts instead: a less direct signal, and a live one, because it is built on data that arrives.

Prove it with history

Then prove the rule would have caught the thing it exists for. Run the expression against the window when the incident happened:

curl -sG http://prometheus.hub.lan/api/v1/query_range \
  --data-urlencode 'query=increase(kube_pod_container_status_restarts_total{cluster="robin"}[15m]) > 3' \
  --data-urlencode 'start=2026-06-29T02:00:00Z' \
  --data-urlencode 'end=2026-06-29T06:00:00Z' \
  --data-urlencode 'step=60s' | jq '.data.result | length'
# a non-zero result over the incident window = the rule is demonstrated, not assumed

If it fires for that period, the alert has been tested without waiting for the next outage.

The principle

An alert is only as real as the data beneath it. Confirm the series reaches the evaluator that will judge it, then validate the rule against a window where the failure genuinely occurred. An untested alert is a belief about the future, and beliefs do not page anyone.

Live in the lab: the restart-count rule was replayed against the original incident window before it shipped.