A metric can be accurate and still be history

This is my homelab, a handful of single-node Kubernetes clusters reporting into one Prometheus. The morning after planned maintenance a dashboard panel went red: workloads not ready. Nothing was wrong. The pods it counted were in phase Succeeded, finished cleanly, and still sitting in the API.

What those particular pods were, I did not record at the time. When I re-checked on 2026-09-06 the only Succeeded pods on the hub were CronJob history: three backup runs per job, kept by each CronJob's history limit. I am not going to claim more than that about the original morning.

Why finished pods linger

Retention is about ownership, not time. A CronJob keeps finished Jobs up to its successfulJobsHistoryLimit and failedJobsHistoryLimit; each Job owns its pods and can carry a ttlSecondsAfterFinished that removes it, and them, after a set period. Deleting an owner takes its dependants with it under the normal cascading deletion, unless the delete asks to orphan them. Below all of that, the controller manager's terminated-pod garbage collector only starts pruning at a count threshold, 12,500 by default, which this hub does not override. That is a count, not a clock. On a small cluster the records stay until something owns their removal.

The panel asked a narrower question than it displayed

count(
  min by (cluster, namespace, pod) (kube_pod_status_ready{condition="true"}) == 0
  unless on (cluster, namespace, pod)
    max by (cluster, namespace, pod) (kube_pod_status_phase{phase="Succeeded"}) == 1
)

kube-state-metrics exports kube_pod_status_phase as one series per pod per phase, with the value 1 on the current phase and 0 on the others. unless matches on labels alone. Without the == 1, the right-hand side contains every pod and the guard subtracts all of them. The result is not zero; it is no result at all. My dashboard carried exactly that version until 2026-09-06, with an or vector(0) on the end that rendered the emptiness as a reassuring zero. It failed towards silence.

What it counts: pod series, not workloads, so one Deployment with three unready replicas is three, and only pods that have a ready-condition series at all. Keep cluster in the identity whenever the data you query holds more than one cluster, which a single Prometheus can; my hub stamps the label on every scrape and on everything remote-written to it, and the scrape labels decide what the right identity is. Use the pod uid too if both sides carry it.

The min by is a policy, not a tidy-up. If two scrapes of the same pod disagree, the not-ready value wins, so a lagging scrape can over-report for one interval. I chose that over max, which would let a stale 1 hide a fresh 0. The fixture for it is in the tests below. Failed, Unknown, Pending and terminating pods all count as not ready, which is what I want from this panel and is not a definition of service availability. When nothing is unready the result is empty. A panel may show that as No data or as 0, and I would rather see No data than hide missing telemetry behind a fallback constant.

Backups are a different question

Backup freshness is its own objective. It does not replace readiness for long-running services, and readiness says nothing about whether last night's snapshot exists.

(time() - max by (cluster, namespace, cronjob) (
  kube_cronjob_status_last_successful_time{cronjob=~"restic-backup.*|etcd-backup"}
)) > 26 * 3600

Namespace is in the grouping on purpose: a fresh job with the same name in another namespace would otherwise hide a stale one. The 26 hours is my policy for a nightly schedule plus the delay I will tolerate, not a universal threshold, and it has to account for the job's own runtime and the timezone the schedule is written in. The hub's live rule groups by cluster and cronjob today; adding namespace is a proposed change here, not a deployed one.

Suspension is a policy decision too, so here is mine. Suspending a backup does not make the last copy any fresher, so this alert keeps firing while a job is suspended, and an informational suspended alert sits beside it to say why. The never-succeeded rule below does exclude suspended jobs, because a job parked before its first run is a choice rather than a failure.

An age check cannot see a job that has never succeeded, because that series does not exist. This is the complete rule, grace period included:

(
  kube_cronjob_info{cronjob=~"restic-backup.*|etcd-backup"}
    unless on (cluster, namespace, cronjob) kube_cronjob_status_last_successful_time
    unless on (cluster, namespace, cronjob) (kube_cronjob_spec_suspend == 1)
)
and on (cluster, namespace, cronjob) ((time() - kube_cronjob_created) > 26 * 3600)

A deleted job needs an expectation that survives the deletion, which means writing the expected identity into a rule and using absent(). That rule is gated on the inventory family itself being present for the cluster, not on up, because a successful scrape proves the fetch and nothing about which metric families were enabled inside it. In this lab the remote clusters forward only the last-success metric, so the inventory rules cover the hub alone; that limit is written into the rule file rather than hidden.

Exporter loss is covered per cluster from an explicit list of the clusters this hub expects, so a healthy exporter elsewhere cannot mask a missing one. A dead exporter does not go quiet at once: its last samples stay selectable for the staleness window, then vanish, and the tests walk through that transition. A succeeded Job proves the job's own exit status. It does not prove the snapshot is complete or restorable. That is a separate check, and I am not claiming to have run one here.

Read before you delete

# Read-only: what is finished, and who owns it.
kubectl get pods -A --field-selector=status.phase=Succeeded \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,FINISHED:.status.containerStatuses[0].state.terminated.finishedAt'

The first version of this post ended with a cluster-wide delete and called it harmless because history regenerates. It is not harmless. Deleting a completed pod removes the API object and, with it, kubectl logs for that container; copies of those logs kept elsewhere survive, but the next run does not recreate the deleted one. If cleanup is wanted, set history limits or a TTL on the Job and let the controller own it, or delete reviewed objects in one namespace after keeping what you need.

Every expression above lives in the site repository as a rule file with promtool unit tests: unready pods, Succeeded and Failed, the missing == 1 regression, disagreeing scrapes of one pod, empty input, the same pod name in two clusters, the same CronJob name in two namespaces, fresh and overdue success, never succeeded, new, suspended, suspended with an overdue success, deleted, one exporter missing while others are healthy, a healthy exporter without the inventory, and a scrape that goes down and comes back with the stale window in between. They pass on promtool 3.12.0 and are in tests/promql.

A count of objects is a count of what the API still holds. Ask what keeps each one alive, filter by value and not only by label, and put the expression under a test before it goes on a wall.

Observed after a maintenance window in this lab; the dashboard guard was corrected 2026-09-06. Revised 2026-09-06.