The server took the new limit. The node kept the old one.

This is my homelab: a Tdarr server on one small Kubernetes cluster and GPU transcode nodes on others, all run by me. Nothing here describes an employer's systems.

On 2026-08-03 I raised one node's GPU worker limit from zero, wrote the change straight into the server's node record through its database endpoint, and read it back. The server said the new value. The node kept drawing no work. I restarted the node's deployment and it started working. Server and node were both Tdarr 2.50.01, the same build they run today.

What I did not do matters as much. I did not use the web interface's per-node worker controls, read the node's own log for the limit it believed it had, or check whether the node ever polls that record. So I know the write persisted and the running node did not act on it. I do not know why.

Stored is not applied

The endpoint I used writes the server's copy of the node record. Reading the value back proves the write persisted, not that the node process is running with it. The limit governs how many workers that process may start, and something has to carry a new limit from the record to the process.

The supported way to change a running node's worker count is the web interface, which Tdarr documents under Nodes and Workers; the startup settings a node reads from its environment are in the container run configuration. Whether my database write bypassed something that path does is a hypothesis. I have not tested it against 2.50.01, and the current documentation describes the supported workflow, not what happened on that day.

There are other candidates I did not rule out: an environment override on the node, a paused state, the worker type, or nothing in the queue being eligible for that node. Node identity I can now rule out. Each node's nodeName is fixed in its manifest and is the record's id; the server holds four records after weeks of pod replacements, so replacing a pod does not create a new one. What does change per job is the working directory name under /temp, and directories left by a pod that died are not reclaimed. That is the lab observation behind my earlier, wrong note that each pod registers as a new node.

This is the write, labelled for what it is. It runs from a pod in the same cluster, where the service name resolves; from a laptop, port-forward the service and point the variable at the local listener:

# Mutation: overwrites the server's record for one node.
CTX="admin@alfred"
NODE_ID="elfastc-b580"      # in 2.50.01 the configured nodeName is the record id
kubectl --context "$CTX" -n media port-forward svc/tdarr 8265:8265 &
TDARR="http://127.0.0.1:8265"
status=$(curl --silent --show-error --max-time 10 -o /dev/null -w '%{http_code}' \
  -X POST "$TDARR/api/v2/cruddb" -H 'Content-Type: application/json' \
  -d "{\"data\":{\"collection\":\"NodeJSONDB\",\"mode\":\"update\",\"docID\":\"$NODE_ID\",
       \"obj\":{\"workerLimits\":{\"healthcheckcpu\":0,\"healthcheckgpu\":1,\"transcodecpu\":0,\"transcodegpu\":2}}}}")
[ "$status" = "200" ] || echo "unexpected HTTP status $status"
# curl exits non-zero on a transport failure; the status check is what insists on 200. The body
# is empty on success and nothing here validates it.

I send the whole workerLimits object. My notes from July say a partial object replaced the nested one and dropped the sibling limits. The server's database module in 2.50.01 is minified, and I have not reproduced that this month, so treat it as a precaution rather than a documented behaviour. Read the record back before writing if anything else might be changing it.

Restarting was the recovery, not the diagnosis

# Mutation: replaces the node pod. Anything encoding on that node is interrupted.
kubectl --context "$CTX" -n media rollout restart deploy/tdarr-node --request-timeout=30s
kubectl --context "$CTX" -n media rollout status deploy/tdarr-node --timeout=180s

Before a restart, pause the node in the Nodes tab so it stops taking new work, and let its workers finish. I have watched the server pull staged files back into the queue after a limbo timeout, but I have not checked what 2.50.01 does with a job killed mid-encode, so do not rely on an interrupted job being requeued. Do not use the database write above as a draining mechanism either; that is the very path whose effect on a running node is in question.

A completed rollout proves a new pod became Ready. It does not prove the limit took effect or that an encode ran.

What I would check now

The server keeps one report per job. It names the node and worker that ran it, records each encoder probe, and stores the full ffmpeg argument list. That is the artefact to correlate against, because it ties a specific job to a specific node and a specific command:

# Read-only, inside the server pod: the newest transcode report for one node, with its argument
# record. Media paths are masked in the output; this is an excerpt of the report, not all of it.
NODE="elfastc-b580"
R="/app/server/Tdarr/DB2/JobReports"
[ -d "$R" ] || { echo "no report directory at $R"; exit 1; }
hit=""
for d in $(ls -t "$R" | head -400); do
  f=$(grep -lsE "Node\[$NODE\]:Worker\[[^]]+\]:\{\"workerType\":\"transcodegpu\"" "$R/$d"/* | head -1)
  [ -n "$f" ] && { hit="$f"; break; }
done
[ -n "$hit" ] || { echo "no transcode report for $NODE in the newest 400"; exit 1; }
echo "report: $hit"
grep -oE '"spawnArgs":\[[^]]*\]' "$hit" | head -1 | sed -E 's#/data/[^"]+#/data/...#g; s#/temp/[^"]+#/temp/...#g'
grep -oE 'Transcode (success|error)' "$hit" | tail -1

Device activity is a supporting signal, not proof. On the NVIDIA nodes I read encoder utilisation with nvidia-smi --query-gpu=utilization.encoder --format=csv; it is GPU-wide, it cannot say which job is using the encoder, and memory in use is not evidence that anything is encoding. The Intel node has no nvidia-smi at all. A falling queue count can mean dispatch, skips, failures or removals.

In August I read a field on the node record, NodeJSONDB.workers, as a live worker inventory, and it was empty while both nodes were encoding. The records on 2.50.01 today do not carry that field at all. I do not know where this version keeps worker state, and I stopped treating that record as a status source.

What I have is a recovery step and an open question. Next time I will use the interface controls first, read the node's log for the limit it thinks it has, and only then reach for the database.

Observed 2026-08-03 and re-checked 2026-09-06 on Tdarr 2.50.01, server and node. Revised 2026-09-06.