Three empty builds before I read the kernel's file

This is my homelab: single-node Kubernetes clusters, one operator, and nothing here describes an employer's systems. The transcode node carries two Intel Arc B580 cards on the kernel's Xe driver, and Tdarr keeps them busy with AV1 work.

When a piece of hardware owes you a number, the habit is to ask whoever speaks for it. The vendor's daemon first, then the standard utility, then the field format every write-up quotes, and when something breaks, the error message's own account of why. Each answers fluently, and each learned its fluency on hardware that is not the card in front of you.

I wanted per-engine busy figures from those two cards while they transcoded. The exporter took five image builds in one evening. The first real row arrived with the fourth, once I read the file the kernel writes off a process using the card; the three before it came back empty because each trusted a translation nobody had checked against the card.

Ask whether the metric exists before you tune for it

The vendor's answer was already running. On 1 July 2026, on this node, a sibling exporter built on Intel's XPU Manager reported power, memory and temperature for both cards, and nothing shaped like an engine or a clock. The instinct at that point is to call the gap a configuration bug and start turning flags.

I did not find a flag that helped. XPU Manager's documentation, when I checked, listed the professional B-series cards as supported devices and not the B580, which would explain the gap. That is my inference from a device list, not a statement from Intel about consumer cards, and a later release may well differ. What I can vouch for is the metrics page I read: no engine, no clock.

The standard utility was a shorter conversation. The intel_gpu_top I had targets the older i915 driver, not Xe. Its Xe-aware successor, gputop from the Debian trixie package, did give real figures by hand: run against a live Tdarr job, it showed the vcs engine at 24.7% and vecs at 17.4%. That build had no scriptable output, only a terminal screen, so the first exporter wrapped that screen in a pseudo-terminal and parsed the table. It worked in a debug pod. Deployed as the real DaemonSet, it reported no GPU clients at all.

I never found out why, and I did not go back to look. A parser of someone else's rendering is a translation of a translation, and the kernel's own per-process file was one layer down.

Treat the error's diagnosis as a false friend

The second build went down that layer. Every process holding a GPU open has a text file per descriptor under /proc/$pid/fdinfo, and the driver writes that process's engine counters into it, in a format of the driver's choosing. Reading another process's copy takes capabilities, so the pod added two, SYS_ADMIN and SYS_PTRACE. The read came back Permission denied.

That message reads like a missing capability, and it points straight at privileged: true. A throwaway privileged probe pod then read another process's fdinfo at once, which looked like proof, and my notes from that night leaned towards a namespace boundary that only full privilege could cross. The error was a false friend: a phrase that means one thing where you learned it and something else here.

Re-tested as one command, finding a process and reading its file straight after, the read worked with the two capabilities the pod already had. One command shortens the gap between finding a process and reading it; it does not close it, because a process can still exit in between, and a process ID can in principle be reused. The notes put the earlier denial down to transcode processes churning between one manual command and the next, and call the privileged probe contaminated by the same timing. I believe that and have not proved it. What the re-test shows is that those two capabilities were enough for this pod on this node, not that both are needed or that nothing smaller would do.

# The container's securityContext as deployed: root inside it, every capability dropped, two added back.
securityContext:
  runAsUser: 0
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  capabilities:
    drop:
      - ALL
    add:
      - SYS_ADMIN
      - SYS_PTRACE
  seccompProfile:
    type: RuntimeDefault

Those fields are not the whole of its privilege. The pod also runs with hostPID, so it can see the host's processes at all, and mounts /sys and /dev/dri from the host. It claims no GPU from the device plugin, because reading activity is not submitting work. The DaemonSet has never run privileged; only the throwaway probe did.

Read the file before you write the parser

Meanwhile the second build's parser had been written before anyone read the file. The write-ups I had leaned on, and the models trained on them, show drm-engine-* lines counting nanoseconds, the form the i915 and amdgpu drivers use. The kernel's own DRM usage-stats documentation describes cycle counters as well; I had not read it. Pointed at a file with no nanosecond lines, that parser raised nothing. It found nothing to parse and reported nothing, cleanly.

The third build misread the silence. A busy figure is the change in a counter between two samples, and the loop took one sample per five-second poll, so a process had to outlive a whole poll cycle to produce a row. I moved both samples inside one scrape, a second apart. The scrape came back clean and empty again. The change suits short jobs and it stayed, but on its own it changed nothing.

The fourth build began where the second should have, with the raw file read off a process that had the card open. Here is that read as I would run it now, one command inside the exporter's own pod, which keeps the gap between finding a process and reading it short. A process that exits in that gap shows up as No such file or directory, which is a different failure from Permission denied and worth keeping apart. It uses FNR rather than NR because every clients file under debugfs carries its own header line; NR quietly lets the second file's header through as a process ID.

# Run while a job is encoding; an idle card has no DRM clients and this prints nothing.
kubectl -n monitoring exec ds/gputop-exporter -- sh -c '
  for pid in $(awk "FNR > 1 { print \$2 }" /sys/kernel/debug/dri/*/clients | sort -u); do
    grep -h "^drm-" "/proc/$pid/fdinfo/"* | sort -u
  done'
# A format check only: on Xe the lines are drm-cycles-* / drm-total-cycles-* pairs, not drm-engine-*.

On Xe that prints pairs such as drm-cycles-vcs and drm-total-cycles-vcs, one pair per engine, and they count GPU clock cycles where the borrowed grammar expected nanoseconds. Same file, same alphabet, different language. Utilisation is the change in the first over the change in the second between two samples, and once the parser spoke that grammar the rows arrived. The fifth build added per-GPU clock frequency from plain sysfs files under tile0/gt0/freq0/ and worked first time.

That loop throws away which file each line came from, so it is for learning the field names, not for counting. The exporter keeps its samples per process, takes both a second apart within one scrape, and skips an engine whose total-cycles counter did not move, rather than dividing by zero. Its error counter, though, counts a process that vanished and a read that was refused alike; the message text tells them apart and the number does not.

The empty page that fooled me is easy to reproduce. This is the healthy fifth image today, with nothing transcoding:

# Read-only: fetch the exporter's own page from inside its pod and keep three of its metrics.
kubectl -n monitoring exec ds/gputop-exporter -- python3 -c \
  "import urllib.request; print(urllib.request.urlopen('http://localhost:8080/metrics').read().decode())" \
  | grep -E '^gpu_(engine_busy_ratio|exporter_scrape_success|exporter_fdinfo_errors)'
# Captured 2026-09-28 with both cards idle: scrape_success 1, fdinfo_errors 0, and not one busy row.

A scrape that succeeds, with zero errors and zero rows. An idle card produces that page, and so did a parser listening for the wrong dialect. The errors counter is there so a failed read does not hide inside an empty result, and it still cannot tell idle from wrong. Only a real job on the card can. It is the same misreading as a scan that finished having walked nothing, seen from the monitoring side.

Read the artefact, not the phrasebook

None of this is special to GPUs. Any interface you have not read on your own system arrives with a phrasebook attached: a vendor's field list, a neighbouring driver's format, the account an error gives of itself. The phrasebook is usually close, and close is the dangerous part. A parser that is badly wrong crashes on its first line. One written for a neighbouring dialect finds nothing it recognises, raises nothing, and ships.

So treat every borrowed format as a hypothesis with a single test that settles it, a raw sample from the real system doing real work. An error that explains itself earns belief only after it survives being run again with less room for the world to change underneath it. And an empty page is not good news until a real workload has filled it at least once.

The artefact will tell you what language it is written in, if you stop reading it aloud in someone else's.

Live on the lab's transcode node: the fifth image reads Xe cycle counters and sysfs frequency for both cards, with two added capabilities and no privileged flag. The first rows came on 2026-07-01 under a real transcode. Checked again on 2026-09-28 with the cards idle, which is why its page was as empty as the ones that fooled me; that idle scrape does not re-check the busy figures, and comparing them against a reference tool on a known workload is still to do.