~/jwright
← All posts
OBSERVABILITY SERIES · 05··revised ·8 min read

A finished scan can have walked nothing

An empty conveyor belt running under a glowing success light while a stack of crates waits untouched beside it

This is my homelab: single-node Kubernetes clusters, one operator, and nothing here describes an employer's systems. The media library is indexed by Tdarr, a transcode server that I drive from scripts through its database endpoint and a handful of actions rather than its web interface.

Errors get investigated. A 200 gets believed, filed and never revisited, and that asymmetry is the most reliable hiding place in any system you did not write yourself.

Mine hid a library scan. The API took the request and answered 200. Its scanner logged a launch and a finish about eighty milliseconds apart, in the right order, and recorded a result of zero files found.

The status code was not lying. It was answering a different question from the one I thought I had asked: whether the request was accepted, not whether the work I wanted had happened. Signing the docket proves a courier came to the dock. It says nothing about what came off the van.

Diagram of two calls to the same scan endpoint. With an empty array as the path the call returns 200 and the scanner logs a launch and a finish about eighty milliseconds apart with zero files found. With the folder as a string it returns 200 and the scanner finds files. Set against the tool's answer, the files on disk and the records in the index are reconciled by comparing path sets, which is how two libraries turned out to be 164 files apart
Diagram scrolls sideways on narrow screens.

The scanner told the truth about nothing

The field that names the folder to walk is called arrayOrPath, and it takes either. In the mode I was using, the additive one that adds new files and leaves existing records alone, an empty array gave it nothing to walk, so the scanner walked nothing and reported that it had finished. I have not tested the empty array against the other scan modes. One of my own scripts uses that form to kick a freshly created library, and I never went back to check what it found.

The eighty milliseconds between the scanner's launch line and its finish line was a clue, not the proof. It is a scanner log interval, not a request time, and I have not established whether this version's endpoint waits for the walk before answering or acknowledges the job and does the work afterwards; an endpoint that does the latter answers quickly whether the walk is empty or not. The evidence was the scanner's own result, zero files found, the index staying exactly where it was, and the different result after passing the folder path. Tdarr's API documentation points at the running instance's own reference rather than a published contract, so the semantics are whatever the installed version does, and I treat the response as an acknowledgement and nothing more.

# The first call is read-only; the second starts a scan. Both run from inside the cluster, where
# the service name resolves; from a workstation, port-forward svc/tdarr in the media namespace and
# point TDARR at the local listener instead.
set -euo pipefail
TDARR="http://tdarr.media.svc.cluster.local:8265"
FOLDER="/data/movies"
# The library id comes from the tool's own database endpoint, selected by folder rather than by
# position: the response is a bare array and nothing promises an order. Exactly one match, or stop.
LIBS="$(curl --silent --show-error --fail --max-time 10 -X POST "$TDARR/api/v2/cruddb" \
  -H 'Content-Type: application/json' \
  -d '{"data":{"collection":"LibrarySettingsJSONDB","mode":"getAll"}}')"
LIB="$(jq -r --arg f "$FOLDER" '[.[] | select(.folder == $f) | ._id]
  | if length == 1 then .[0] else error("expected one library for \($f), found \(length)") end' <<<"$LIBS")"

# Mutation: this starts a scan. With an empty array in the additive mode it walks nothing, which is
# the whole point of running it. The body is built by jq so the id is never string-interpolated.
jq -n --arg id "$LIB" '{data:{scanConfig:{dbID:$id,arrayOrPath:[],mode:"scanFindNew"}}}' \
  | curl --silent --show-error --fail --max-time 30 -o /dev/null -w '%{http_code}\n' \
      -X POST "$TDARR/api/v2/scan-files" -H 'Content-Type: application/json' -d @-
# 200. The scanner log shows a launch and a finish and zero files found.

The correction is a type change. Pass the folder as a string and the same endpoint finds files.

# Mutation: this one scans for real. Additive mode, so existing records and their transcode
# history survive it.
jq -n --arg id "$LIB" --arg f "$FOLDER" '{data:{scanConfig:{dbID:$id,arrayOrPath:$f,mode:"scanFindNew"}}}' \
  | curl --silent --show-error --fail --max-time 300 -o /dev/null -w '%{http_code}\n' \
      -X POST "$TDARR/api/v2/scan-files" -H 'Content-Type: application/json' -d @-
# 200 again. This time the scanner log reports the files it found.

Same status code, same pair of log lines, a different count on the result line. The status was never the variable. The input was, and nothing in the exchange said so.

Compare the inventory, not the totals

The deeper mistake was trusting the tool's own tally. It reported a library size, the number looked about right, and nothing was ever set against it. A tool's self-reported completeness is a claim about itself, checked by itself.

When I finally counted from outside, on 2026-08-03, the movie library had 265 files on disk against 236 records, and the shows library 5082 against 4947: 29 and 135, 164 across the two. Folder watching was off on both at the time, so nothing was noticing new media on its own either. It is on today, which changes how files arrive and changes nothing about whether anyone checks.

Two totals are a screen, not a proof. They match when one path is missing and one record is stale, and they differ for reasons that are not missing work: a file that landed during the walk, a directory the walker could not read, a name the filters disagree on. The record of my own repair shows it: after the corrected rescan the movie index read 264 against 265 on disk, one short, and a matching-looking pair of totals would have hidden that remainder. So the check has two stages. The totals say whether to look. The path sets say where.

# Read-only. Both sides of the comparison: the files on disk under the library folder, walked by
# find inside the server container, and the paths the index holds for that library. The walk is
# NUL-separated so a filename can never split into two records, and under pipefail a walk that hits
# an unreadable directory fails the check instead of returning a smaller number. LIB, FOLDER and
# TDARR are the values from the block above; CTX is whichever kubeconfig context holds the cluster.
# Run it as a script: it sets pipefail for the reason above.
set -euo pipefail
CTX="$(kubectl config current-context)"
kubectl --context "$CTX" -n media exec deploy/tdarr -c tdarr -- \
  find "$FOLDER" -type f \( -iname '*.mkv' -o -iname '*.mp4' \) -print0 | sort -z > disk.z
# The index side, filtered to the same library and the same extensions, one NUL-terminated path
# per record. The collection holds every library's records, so the filter on DB is not optional.
curl --silent --show-error --fail --max-time 120 -X POST "$TDARR/api/v2/cruddb" \
  -H 'Content-Type: application/json' \
  -d '{"data":{"collection":"FileJSONDB","mode":"getAll"}}' \
  | jq --raw-output0 --arg id "$LIB" '.[] | select(.DB == $id)
      | select(.file | test("\\.(mkv|mp4)$"; "i")) | .file' | sort -z > index.z
printf 'disk %s  index %s\n' "$(tr -cd '\0' < disk.z | wc -c)" "$(tr -cd '\0' < index.z | wc -c)"
comm -z -3 disk.z index.z | tr '\0' '\n'
# First column: on disk and not in the index. Second column: in the index and not on disk.
# Run on 2026-09-18 this read 345 against 344 and printed one path, in the first column.

The same gap pointed the other way here once before: a pipeline that reported failure for a month while every artefact landed. That post argues the artefact outranks the status, and I still think so. This one is about the case where the artefact is exactly what you cannot see from where you are standing, so you have to go and count it.

Know which mode clears the records

The endpoint carries a mode that is not additive. scanFindNew adds new files and leaves existing records and their transcode history alone. scanFresh resets the library's file records to zero and then rescans.

Pair that second mode with a call that walks nothing and the reset lands, the rescan finds nothing to put back, and both calls answer 200. I have not run that pairing and do not plan to. Knowing which mode is additive is the difference between a slow drift and an instant hole where the records were.

Reconcile the inventory

Every tool reports on itself, and the report is written by the same code that did or did not do the work. A success code tells you the input was accepted; it does not tell you the input was the one you meant, and an empty input is the easiest thing in the world to accept. What is left is the inventory the tool did not author: the files as the filesystem holds them, set path by path against the records the tool holds, with the same filters on both sides. Two totals that ought to match and do not are a reason to look. Two path sets that differ are the answer.

A signed docket is not a stocktake.

Observed in this lab on 2026-08-03 against Tdarr 2.50.01, the version still running here. The reconciliation above was run again on 2026-09-18 and found one file on disk that the index did not hold. Nothing schedules it, so a drift is still found by asking.