#Tag
Posts tagged "observability"
Three empty builds before I read the kernel's file
A per-engine GPU exporter for two consumer Intel Arc cards took five builds in one evening, and the three before the first real row came back empty because each trusted a reading borrowed from somewhere else. The row arrived only after the kernel's own file was read off the working card, which is where any parser for an interface you have not read yourself should start.
A finished scan can have walked nothing
A library scan answered 200 and its scanner logged a launch and a finish about eighty milliseconds apart with zero files found, because one parameter was an empty array where a folder path belonged. The tool had done exactly what it was asked. Only an inventory taken outside it, set against its own records, showed two libraries 164 files short.
A metric can be accurate and still be history
After a maintenance window my not-ready panel went red over a cluster where nothing was wrong. It was counting Succeeded pods still held in the API. The fix is an exclusion filtered by value for readiness and a separate freshness check for backups, and both expressions now have unit tests.
Two clusters, one view, and the label that decides whether it works
Forwarding metrics from every cluster to one Prometheus does not give you a fleet view; it gives you one Prometheus full of series that cannot say where they came from. The whole design turns on a single label, and the part everyone misses is that the hub's own scrapes never pass through the forwarding config that stamps it.
The alert that could never fire
A readiness alert that passed review was watching for a series the remote cluster's collector filters out before it ever leaves. An alert you never test is a belief, not a control.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.