#Tag
Posts tagged "prometheus"
fsGroup is a request, not a guarantee
A non-root pod crashlooped on permission denied with a textbook securityContext, because the volume plugin behind a host-directory-backed claim carries no ownership pass and nothing says so. The field validated, the ownership change never happened, and only the directory reached through the application's own mount could settle it.
A metric can be accurate and still be history
After a maintenance window my not-ready panel went red over a cluster where nothing was wrong. It was counting Succeeded pods still held in the API. The fix is an exclusion filtered by value for readiness and a separate freshness check for backups, and both expressions now have unit tests.
Two clusters, one view, and the label that decides whether it works
Forwarding metrics from every cluster to one Prometheus does not give you a fleet view; it gives you one Prometheus full of series that cannot say where they came from. The whole design turns on a single label, and the part everyone misses is that the hub's own scrapes never pass through the forwarding config that stamps it.
The alert that could never fire
A readiness alert that passed review was watching for a series the remote cluster's collector filters out before it ever leaves. An alert you never test is a belief, not a control.