Every build succeeded. The dashboard said they all failed.

The dashboard showed red. It had shown red for every run in a month. Underneath, every single one of those builds had completed: images built and pushed, deploy branches created, artefacts sitting exactly where they should be.

The cost of believing the dashboard was a three-post release that had been ready to ship for a fortnight. Nobody merged it, because the pipeline said it failed.

Thirteen minutes, every time

Real failures are fast. A broken build dies at the failing step - seconds in, maybe a minute. These runs sat in progress for thirteen minutes and then flipped to failed, all of them, with the same suspicious regularity.

That shape isn't a build failing. It's a build finishing and its result never arriving, followed by something timing out and recording an assumption. The runner logs confirmed it: every step green, job succeeded, and nothing after that reached the server.

The artefact cannot lie about existing

Once the reporting channel is suspect, the status is worthless in both directions, and the way out is to check for the thing the pipeline was supposed to produce. Two commands outrank the dashboard:

# did the image land, and what digest is it really at?
skopeo inspect --format '{{.Digest}}' docker://registry.example.lan/site/web:0826R2

# did the pipeline push its deploy branch?
git ls-remote origin 'refs/heads/deploy/*'

An image with the expected digest and a deploy branch on the remote are facts on disk, immune to whatever the status API believes.

That became the working rule while the bug persists: for this pipeline, red means nothing, and the question is whether the artefact exists.

The messenger is a component too

The pipeline had two jobs: do the work, and report the work. Only the first was ever tested. The callback that carries a result to the dashboard is a system of its own - it can break independently of the thing it reports on, and when it does, what it emits is still a well-formed status that reads like news. Nothing in the pipeline's own checks could catch that, because from where the runner stood, every job had succeeded and every duty was done.

One thing failed all month

Did anything actually fail that month? One thing: the messenger. The images were in the registry, the deploy branches were on the server, and the release shipped the same afternoon somebody checked for the artefacts instead of the status. Status is a claim. The artefact is the evidence.

Still true in the lab: red on that pipeline means nothing until the deploy branch is checked.