fsGroup is a request, not a guarantee

This is my homelab: single-node Kubernetes clusters on Talos, one operator, and nothing here describes an employer's systems. Persistent storage on every cluster is a local-path provisioner, which hands a workload a directory on the node dressed as a PersistentVolume.

Prometheus would not start. Clean, repeatable crashloop: permission denied opening its active query log. The pod runs non-root, as it should. The spec carries fsGroup, and the documentation says volumes that support ownership management are made owned and writable by that group before the container starts. The claim was bound, the pod was scheduled, everything was green except the one process the namespace exists for.

We read securityContext as configuration: set the field, and the platform makes it true. Some of those fields are not settings. They are requests, slips of paper handed to whichever component has to do the actual work, and nobody tells you when a request goes in the bin.

Find the component that quietly declined

The API server checks that the slip is filled in correctly. The component that has to act on it is the volume plugin behind the claim: at mount time the kubelet passes the requested group to that plugin, and a plugin that supports ownership management applies it. The security context documentation says exactly that, and leaves which plugins those are to each plugin.

Upstream's host-path plugin is one that does not. Its mount setup validates the path and its type and returns; there is no ownership pass in it. The field is still not ignored, which is a distinction worth keeping: fsGroup also joins the group to every process in the pod, and that part happened. The ownership request is the part that had nowhere to go.

Nothing was rejected, because the field is valid. No event was recorded, because nothing failed and a step was merely absent. The pod carried no condition about it, and the logs at default verbosity said nothing. The first component to mention a problem was the application itself, hitting the filesystem and being told no. That is a genre of failure rather than a one-off: an absent step leaves no trace anywhere except in the thing that needed it.

A storage class name does not tell you which plugin you got. The local-path provisioner creates either hostPath or local volumes depending on how it is configured, so the only place the answer lives is the bound PersistentVolume.

# Read-only. Which volume type is actually behind the claim. The answer is in the bound PV, not in
# the storage class name.
PVC="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"
PV="$(kubectl -n monitoring get pvc "$PVC" -o jsonpath='{.spec.volumeName}')"
kubectl get pv "$PV" -o jsonpath='{.spec.hostPath.path}{" "}{.spec.local.path}{" "}{.spec.csi.driver}{"\n"}'
# Here: a hostPath path, no local path, no CSI driver. The ownership request went to a plugin
# that has no code to act on it.

Chown through the mount the application uses

If the platform will not change the ownership, a root init container can. One chown of the volume and done. Except the second trap is meaner than the first.

The operator mounts the data volume with a subPath. When a container mounts a subPath, the kubelet resolves it and bind-mounts that directory instead of the volume root, so the application's data path is a different directory from the one an init container sees when it mounts the bare volume. Changing the owner of that root touches the root. An existing child keeps its owner unless the change is recursive, and a directory the kubelet creates for a subPath is created as root.

# prometheusSpec.initContainers on the kube-prometheus-stack release, as deployed. The volume name
# is the StatefulSet's claim template name; the subPath is the one the operator mounts for the
# server. Runs as UID 0, which the namespace's Pod Security level has to admit.
initContainers:
  - name: init-chown-data
    image: busybox:1.37
    command:
      - sh
      - -c
      - chown -R 1000:2000 /prometheus
    securityContext:
      runAsUser: 0
      runAsNonRoot: false
    volumeMounts:
      - name: prometheus-kube-prometheus-stack-prometheus-db
        mountPath: /prometheus
        subPath: prometheus-db

The load-bearing line is the last one. With the same subPath the server uses, the kubelet resolves both mounts to the same directory and the ownership change lands where the write will happen. That fixed this deployment. It does not mean a recursive change through the volume root can never reach a child; it means the safe recipe is to mount what the application mounts and not reason about the tree from outside it.

I checked the pieces of that on a disposable claim on the same storage class on 2026-09-18. A root pod on the volume root created a child directory and changed the root's owner: the child was unchanged. It changed the owner recursively: the child changed. Then a second pod mounted that child as a subPath, and the directory it was given was root-owned, empty, and on a different filesystem from the volume root; a file created through the subPath never appeared under the root, and the child created under the root never appeared through the subPath. On this cluster the two mounts are not parent and child. They are two directories, and whatever you chown through one, the application reads the other. Why the kubelet resolved the subPath there is a question about the node's configuration, mine rather than the platform's, and it is not the one this post answers.

Two costs come with the recipe. The init needs the namespace to admit a container running as UID 0, so the monitoring namespace carries the privileged Pod Security level; that is an exception scoped to one namespace that also runs the node exporter, not a licence to label every namespace that way because one init wants root. And a recursive chown walks every file in the volume at each pod start. This volume is small enough that I accept it, and the init only runs when the pod is created.

Check it through the mount the application uses

What changed after this was where I look first. A spec is a statement of intent; the mounted directory is the fact, and it answers a question the documentation cannot.

The server image here is a distroless build with no shell, so there is nothing to exec into; I listed its filesystem to be sure before writing that sentence. Bring a short-lived pod that has a shell, mount the same claim through the same subPath, and read the numeric owner.

# Creates a temporary diagnostic pod; the data mount is read-only and it removes itself on exit.
# Mounts the SAME subPath the server uses, so it reads the directory the application opens, not the
# volume root. The claim is ReadWriteOnce on a single node, so a second pod may mount it; a
# ReadWriteOncePod claim would refuse.
kubectl -n monitoring run subpath-check --rm --attach --restart=Never --image=busybox:1.37 \
  --overrides='{"spec":{"automountServiceAccountToken":false,"volumes":[{"name":"db","persistentVolumeClaim":{"claimName":"prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}}],"containers":[{"name":"subpath-check","image":"busybox:1.37","command":["ls","-lnd","/prometheus"],"volumeMounts":[{"name":"db","mountPath":"/prometheus","subPath":"prometheus-db","readOnly":true}]}]}}'

Numeric, because a name resolved inside a container tells you only about that container's own user database. Through the subPath, because that is the directory the application opens, and the node's view of the volume root can show one owner while the directory the application reaches shows another. That mismatch is the tell. If the numbers are not the ones the field promised, no amount of re-reading the manifest will make them so.

There is a sting in the tail, and it cost a second evening two and a half months later. After a node reboot on 2026-08-27 the directory the server opened was root-owned again and dated to the reboot, while the init container showed as completed, and the application restarted into the same permission error fifteen times. A crash-looping container restarts in place, and a container restart does not re-run the pod's regular init containers; the kubelet runs them again when the pod itself is recreated, and in the other pod-level restarts the init container lifecycle describes. How the reboot left that directory root-owned is an observation, not a mechanism I have proven.

# Disruptive: this deletes the running server pod. The StatefulSet controller replaces it, and the
# replacement runs its init containers again. A container restart never does.
kubectl -n monitoring delete pod prometheus-kube-prometheus-stack-prometheus-0

The kill-test for the fix belongs on a throwaway claim, not on the live database: chown the subPath directory back to root there, delete the pod, and watch whether the replacement's init repairs it.

Check the consuming mount

securityContext fields are promises somebody else has to keep, and for storage that somebody is free to decline without a word. The API accepting your field means only that the request parsed. For any field whose effect lands outside the API server, ownership and mounts and whatever else the node has to carry out, go and stand at the mount the application consumes and read the owner there. A slip of paper is not a key. A field that validates is not a field that happened.

Observed in this lab on 2026-06-14, again after a node reboot on 2026-08-27, and checked piece by piece on a disposable claim on 2026-09-18 (Talos v1.13.9, kubelet v1.35.8, local-path hostPath volumes). The ownership check above was re-run on 2026-09-17 and the directory reads 1000:2000. Chowning through the same subPath is the house recipe for making that directory writable; where that directory lives on the node is a separate matter, and not this post's.