Root in the Pod, Not on the Node: Validate Kubernetes User Namespaces

A Pod can start successfully with spec.hostUsers: false and still be a bad production candidate. The kernel may support user namespaces while one mounted filesystem does not. A stateless container can prove UID remapping while a persistent volume fails on first write. Admission can accept the Pod while its service account still has excessive Kubernetes API rights. A dashboard can stay green while nobody has captured the host-side mapping that the security review actually asked for.

That is the operating problem this playbook addresses. Kubernetes user namespaces became stable in Kubernetes v1.36, but stable feature state is not the same as workload compatibility. The version-pinned Kubernetes: User Namespaces, v1.36 snapshot (accessed September 15, 2026) marks the feature stable and still makes use an individual Pod decision through spec.hostUsers: false; the Kubernetes: User Namespaces, living concepts page (accessed September 15, 2026) says the same at the research cutoff.

The goal is therefore not “enable user namespaces everywhere.” It is to produce three artifacts per workload: mapping proof, storage compatibility evidence, and a residual-permission assessment. Everything that follows is a proposed operating model unless Kubernetes or Linux behavior is explicitly attributed to a source. The examples are synthetic and were not executed against a real cluster.

Define the isolation claim and freeze the evidence before changing manifests

Start by naming the exact claim. For this rollout, the claim is narrow: when a Linux Pod opts into a user namespace, a process that appears as UID 0 inside that Pod is mapped to an unprivileged host identity, and the workload continues to meet its required storage and application behaviors. Kubernetes documents that root inside the container can map to a different, non-root user on the host and that capabilities granted within the Pod's user namespace are void outside that namespace.

That claim does not say the service account is least-privileged, network traffic is constrained, secrets are safe, or the workload has no dangerous Linux capabilities inside its namespace. Those are separate controls. This article deliberately stays narrower than Refonte Learning's broader Kubernetes production-hardening foundations, which cover cluster, network, identity, supply-chain and operational hardening as a whole.

Before editing a manifest, freeze the candidate environment. A control-plane version alone is insufficient because user-namespace success depends on the worker kernel, Container Runtime Interface (CRI) implementation, Open Container Initiative (OCI) runtime, kubelet filesystem, volume filesystems and storage path. The official prerequisites explicitly span those layers.

Use a version-controlled evidence record. The following fields are a proposed local inventory, not a Kubernetes API object:

Treat “unknown” as a failed gate, not an invitation to infer. Managed Kubernetes products can abstract the node image, runtime configuration or storage implementation. Unless the provider documents the exact combination or you can probe the actual node pool, record support as unverified. Keep the same discipline used for infrastructure ownership and code review: the artifact that changes behavior should have an accountable owner, a diff, an acceptance record and a rollback reference.

The baseline matters as much as the candidate. Capture the unchanged workload's scheduling, startup, health checks, file ownership view, read/write behavior, restart behavior and service indicators before adding hostUsers: false. Without that baseline, a later permission failure becomes an argument about whether the application was already broken.

Local stop rule: do not start rollout if the node/runtime inventory is incomplete, if any required volume path is unclassified, or if the known-good manifest cannot be restored from version control.

Establish kernel, runtime and filesystem eligibility on the candidate node pool

Use an isolated staging node pool first. The version-pinned v1.36 documentation requires Linux support for idmapped mounts on the filesystem containing the kubelet Pod directory and on all filesystems used by Pod volumes. It calls Linux 6.3 the practical minimum because tmpfs gained idmapped-mount support there, which matters because Kubernetes commonly uses tmpfs for items such as mounted service-account tokens and Secrets. The same page lists btrfs, ext4, XFS, FAT, tmpfs and overlayfs among popular filesystems supporting idmapped mounts with Linux 6.3.

For runtimes, that v1.36 snapshot lists runc 1.2+ and crun 1.9+ as OCI runtimes with support, recommending crun 1.13+; for CRI runtimes it lists containerd 2.0+ and CRI-O 1.25+. These are documented baselines, not a certification that any vendor's packaging, kernel backports, snapshotter, CSI stack or node image works together.

The Linux kernel's own Linux kernel: Idmappings documentation (living documentation; accessed September 16, 2026) explains why idmapped mounts matter operationally: they let the same filesystem objects be exposed through a mount with translated ownership, with the translation localized to that mount rather than permanently rewriting on-disk ownership. The kernel documentation specifically contrasts this with recursive chown, whose ownership changes are global and permanent.

Kubernetes also documents an explicit storage limitation at the cutoff: NFS volumes cannot be mounted in a user-namespace Pod because the Linux NFS client does not support idmapped mounts. The same limitation section disallows raw block volumeDevices, and forbids combining hostUsers: false with hostNetwork: true, hostPID: true or hostIPC: true.

That NFS statement is useful for a failure fixture because it gives you a known-incompatible path without damaging production data. It should not be generalized into “network storage never works.” The decision is about the actual filesystem and mount path presented to the Pod. Likewise, a successful emptyDir or stateless probe is not evidence that a CSI-provisioned persistent volume works.

The living Kubernetes page at the cutoff was last modified September 9, 2026 and still says the feature is stable since v1.36, while the version-pinned v1.36 page is explicitly a static, no-longer-maintained snapshot. Use the pinned page to define the chosen baseline and the living page to detect later documentation changes; do not silently merge behavior from different versions.

A practical eligibility sequence is:

1. Inventory the node pool. Record kernel, CRI, OCI runtime, kubelet Pod-path filesystem and node image build.

2. Inventory every mount. Include persistent volumes, emptyDir, Secrets, ConfigMaps, projected volumes and any application-specific mounts. Do not inspect only the PVC that appears “important.”

3. Reject manifest-level incompatibilities early. Host PID/network/IPC sharing and raw block devices should fail the candidate gate before runtime testing.

4. Run a probe on the exact node image. A documented minimum is necessary evidence; the probe is what tests the assembled system.

5. Keep storage qualification per class and per workload. A driver upgrade, filesystem change or new mount should invalidate the relevant compatibility record until retested.

Local stop rule: block the workload when a required filesystem cannot be shown to support the needed idmapped mount, when the Pod requires a prohibited host namespace or raw block device, or when the platform team cannot reproduce the node/runtime combination in a controlled test pool. Do not upgrade production nodes merely to satisfy this article's example baseline; treat a node upgrade as its own infrastructure change with its own rollback plan.

Review hostUsers: false through admission, then prove the mapping on both sides

The manifest change should be intentionally boring. hostUsers: false opts an individual Pod into a user namespace; it is not a cluster-wide declaration that every Pod is isolated this way. The kubelet selects host UID/GID mappings and guarantees that two Pods on the same node do not use the same mapping. Kubernetes also states that fields such as runAsUser, runAsGroup and fsGroup refer to identities inside the container.

For the v1.36 baseline, Kubernetes documents a default in-Pod UID/GID range of 0–65535. By default, kubelet assigns Pod host IDs above that range; a node operator can instead configure a subordinate UID/GID range for the kubelet account. Since Kubernetes v1.33, KubeletConfiguration.userNamespaces.idsPerPod can change the number of IDs assigned per Pod, with 65,536 as the documented default. Capture that node configuration if it is customized. These facts explain allocation mechanics; they still do not justify predicting the numeric host range a recreated workload will receive.

A minimal candidate diff should preserve the workload's existing security context, image digest, service account, volumes and policy labels. This synthetic example is not executed and uses placeholders deliberately:

apiVersion: v1
kind: Pod
metadata:
  name: worker-userns-candidate
  namespace: <DISPOSABLE_TEST_NAMESPACE>
spec:
  hostUsers: false
  serviceAccountName: <EXISTING_SERVICE_ACCOUNT>
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: worker
      image: <REGISTRY>/<IMAGE>@sha256:<PINNED_DIGEST>
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      command: ["<EXISTING_COMMAND>"]
      args: ["<EXISTING_ARGS>"]

Do not add privileged mode, host namespaces, a broad capability set, a new service account binding or an unrestricted volume simply to force this candidate to start. If admission rejects the Pod, the rejection is evidence about the policy boundary. Changing policy to make the test green destroys that evidence.

User namespaces interact with Pod Security Standards (PSS) in a version-dependent way. Kubernetes documents that, for Linux Pods using user namespaces, checks on runAsNonRoot and runAsUser are relaxed even under Baseline or Restricted because root in the Pod is not mapped to a privileged host user. Baseline also relaxes certain procMount validation, while Restricted still requires the default or empty ProcMount. Other Restricted controls remain separate, including allowPrivilegeEscalation: false, seccomp requirements and dropping ALL capabilities except the permitted NET_BIND_SERVICE add-back.

Next, prove the mapping. The official Kubernetes: “Use a User Namespace With a Pod” (accessed September 15, 2026) has the right starting observations: inside the Pod, inspect /proc/self/ns/user and /proc/self/uid_map, then compare namespace information on the host. The task's sample output is illustrative only; your evidence must record your own observed values.

Inside the authorized staging Pod:

set -eu
id
readlink /proc/self/ns/user
cat /proc/self/uid_map
cat /proc/self/gid_map

On the node, first obtain the container process ID using your approved, runtime-specific inspection procedure. Do not invent a portable PID-discovery command: containerd, CRI-O, managed services and operational access models differ. Once an authorized reviewer has the correct PID, read-only Linux observations can include:

set -eu
PID="<VERIFIED_CONTAINER_PROCESS_PID>"
test -r "/proc/${PID}/uid_map"
ps -o pid,ppid,user,uid,group,gid,comm -p "${PID}"
readlink "/proc/${PID}/ns/user"
cat "/proc/${PID}/uid_map"
cat "/proc/${PID}/gid_map"

Record evidence, not an expected magic number:

Kubernetes' task says the namespace seen inside the container should differ from the host and uses uid_map comparison as the mapping check. The concepts page adds an important scheduling property: the kubelet chooses non-overlapping mappings for different Pods on the same node. Do not turn that into a promise that a particular numeric host mapping persists after deletion and recreation.

Run a two-Pod test and a recreation test. Schedule two candidates onto the same staging node, capture each mapping, verify the mappings are distinct, then delete and recreate one Pod through its controller and capture the new evidence. Acceptance is about isolation and correct workload behavior, not numeric stability.

Local stop rule: no host-side correlation means no mapping proof. A screenshot of id inside the container alone is insufficient because “root inside” is exactly the condition user namespaces are designed to reinterpret.

Validate storage semantics and keep capabilities, tokens and RBAC as independent gates

Storage validation is where a large fraction of rollout optimism should be spent. Kubernetes v1.36 explains that runAsUser, runAsGroup and fsGroup continue to refer to container identities, and that supported idmapped volume mounts let Pods switch user namespaces without permanently changing volume ownership. The kernel documentation similarly describes idmapped mounts as a temporary, mount-local ownership translation rather than a global ownership rewrite.

Build a disposable dataset that resembles the application's real permission model. For a stateful test, include an existing file owned by the expected application UID/GID, a group-writable directory if the application relies on shared group access, a new-file creation path, a rename/replace path, and a checksum-protected file that must survive Pod recreation.

For the incompatible fixture, NFS is a defensible synthetic choice at this cutoff because Kubernetes explicitly says NFS volumes cannot be mounted into a user-namespace Pod due to Linux NFS client limitations. Capture the actual event and runtime error from your environment rather than pasting documentation's example error into an incident record. Kubernetes warns that failure details depend on the container runtime.

Never respond to a candidate's permission failure with chown -R against a production volume. That changes persistent ownership globally, can break the known-good workload, can be extremely expensive on large trees, and defeats the purpose of testing idmapped mount semantics. The Linux kernel documentation explicitly describes recursive ownership changes as global and permanent, whereas idmapped-mount ownership translation is localized to the mount.

Now split the security review into orthogonal questions. A correct UID map is not evidence that the process lacks dangerous capabilities inside its namespace, that its token is unnecessary, or that Role-Based Access Control (RBAC) is least-privileged. The Kubernetes: “Using RBAC Authorization” (accessed September 15, 2026) warns that broad grants can give service accounts unnecessary and potentially escalating API access, and recommends application-specific service accounts with fine-grained bindings.

The KubeCap authors' 2026 study is relevant only as evidence that capability configuration is often neglected in the three open-source datasets they examined: they report 74.67% of projects in those datasets lacked capability configurations. That is not a user-namespace adoption rate, not a production-cluster estimate, and not evidence that your workload needs a particular capability set. The paper's scope supports keeping capability minimization as its own review gate. Yuhao Liu et al.: “KubeCap: A Framework for Capability Minimization in Kubernetes via Static Analysis and LLM-Assisted Rule Inference,” submitted August 27, 2026.

This separation also aligns with layered cloud-security controls: identity, access, runtime and infrastructure controls solve different failure modes and should not be collapsed into one “hardened” label.

Use an authorization check that does not require reading a real Secret. Kubernetes documents kubectl auth can-i with service-account impersonation; the caller performing the check must itself be authorized to impersonate that identity.

set -eu
NS="<DISPOSABLE_TEST_NAMESPACE>"
SA="<WORKLOAD_SERVICE_ACCOUNT>"
SECRET="<SYNTHETIC_SECRET_NAME>"

kubectl auth can-i get "secrets/${SECRET}" \
  --as="system:serviceaccount:${NS}:${SA}" \
  --namespace="${NS}"

For the required overbroad-permission drill, create a disposable namespace fixture in which the workload service account is intentionally granted read access to a synthetic Secret. Do not put real credentials in it. Verify that the authorization check says the action is allowed, then remove the overbroad binding and verify it is denied. Leave hostUsers: false unchanged throughout. The lesson is the separation: UID remapping should continue to pass both before and after the RBAC correction.

Local stop rule: mapping success cannot override a failed RBAC review, unexplained capability requirement or storage mismatch. Each gate has its own owner and can independently block promotion.

Run the mixed-workload acceptance exercise and classify failures by layer

Use two synthetic workloads on the isolated staging pool. The first is a low-risk stateless worker with no persistent volume. The second is a stateful job that reads and writes a shared test volume. For each, keep an unchanged baseline and a candidate that differs only where required to set hostUsers: false.

The stateless worker goes first because it narrows the kernel/runtime/admission question without hiding behind storage. Run baseline and candidate through scheduling, image pull, init containers, startup probes, readiness, normal processing, termination and recreation. Capture mapping evidence on the candidate. Any arbitrary canary percentage would be a local choice, not a Kubernetes guarantee, so the initial staging scope should be defined by blast radius and owner coverage rather than a copied number.

For the stateful job, use a disposable compatible PVC with representative ownership and sharing semantics. Run baseline and candidate against equivalent datasets, compare checksums and application outcomes, recreate the candidate Pod, then repeat. Add the documented-incompatible NFS fixture as a separate negative test. The NFS result should block that path; it does not justify weakening admission or changing production ownership.

The worked decision record should stay explicit about expected versus observed results:

When something fails, classify it before fixing it. Do not call every red Pod “user namespace incompatibility.”

Kubernetes' user-namespace page includes an example idmapped-mount failure but explicitly notes runtime error details can vary. Therefore, your runbook should require exact observed text from the local event/log record and should never fabricate a vendor error string.

A useful failure drill is intentionally simple. Run the stateful candidate against the incompatible test storage. Confirm the rollout controller or human gate stops there. Confirm nobody edits Pod Security labels, adds privileged mode, injects hostPID, or broadens capabilities to “see if it helps.” Restore the compatible test fixture or the baseline manifest, verify the data checksum, and record the failure as a storage eligibility result.

A second drill uses the intentionally overbroad service-account permission. Mapping and storage can pass while authorization fails the security gate. Remove the binding, re-run kubectl auth can-i, and verify the workload still functions. This demonstrates why “root is remapped” and “the workload cannot read Secrets” are different statements. Kubernetes' RBAC guidance explicitly favors specific resources and verbs over wildcards and warns that broad service-account grants create unnecessary API access.

Local stop rule: uncontrolled rollout stops on any unexplained admission rejection, runtime setup error, required I/O failure, checksum change, lost mapping evidence, policy weakening, or newly required host privilege. The accountable layer owner must either produce evidence for a supported fix or keep the workload on the baseline.

Observe rollout health, rehearse recovery and promote only with an evidence pack

Availability is necessary but not proof of isolation. The v1.36 page documents two kubelet Prometheus metrics for user namespaces: started_user_namespaced_pods_total, counting attempted user-namespaced Pod starts, and started_user_namespaced_pods_errors_total, counting errors creating them. Those metrics can tell you whether candidate creation is happening and failing, but they do not prove which host UID/GID mapping a particular workload received.

Build a bounded-cardinality rollout view around operational questions rather than dumping sensitive process details into labels:

The dashboard should follow normal observability principles: connect symptoms to diagnostic evidence, but keep the evidence categories distinct. A green service-level objective does not establish a UID map, and a correct UID map does not establish healthy application behavior.

Recovery starts with the workload manifest, not the node runtime. Keep the known-good baseline in version control and make the user-namespace candidate a reversible workload revision. If a candidate fails a gate, restore that baseline for the approved test scope, recreate the affected Pod through its controller, and verify the same data path the baseline used before the test.

A proposed recovery checklist is:

• Freeze further candidate promotion and preserve Pod events, kubelet/runtime logs, mapping evidence and storage test results.

• Identify whether data changed. Compare the pre-test and post-test checksums or application-level integrity marker.

• Restore <KNOWN_GOOD_GIT_REF> rather than editing the live Pod by hand.

• Recreate only the approved test workload; do not drain or rebuild unrelated nodes unless the failure is proven to be node infrastructure.

• Verify baseline read/write behavior and application health on the original data path.

• Confirm admission and RBAC policy were not relaxed during troubleshooting.

• Record the candidate as blocked, with the layer owner and next evidence required.

• Treat any node-image, kernel, containerd/CRI-O or OCI-runtime rollback as a separate infrastructure change.

The storage recovery boundary is strict: never recursively chown a production volume as an improvised user-namespace fix. Besides making the test irreversible, that can break the old workload's ownership expectations. The kernel's idmapped-mount design exists precisely to make ownership translation mount-local rather than globally rewriting the filesystem.

Use established monitoring and logging practices to preserve the operational timeline, including events, kubelet/runtime errors and application symptoms, without logging service-account tokens, Secret payloads or unnecessary per-process identity data.

Promotion should be per workload, not per cluster. A proposed evidence-based status model is eligible, validated, adopted, blocked, or exception. “Feature stable” is not a promotion status. An exception should name the reason, owner, compensating control, expiration/review date and evidence needed to retire it.

A practical sequence is inventory → eligibility → stateless probe → mapping proof → stateful volume test → RBAC/capability review → recovery drill → limited adoption → broader workload-by-workload adoption. Re-open a workload's evidence when the node image, kernel, CRI/OCI runtime, filesystem, CSI driver, security policy, service account or material application permission model changes.

Build a reviewable validation portfolio and answer the remaining user-namespace questions

The useful output of this work is not a slide that says “Kubernetes user namespaces enabled.” It is a small portfolio that another engineer can reproduce: the baseline and candidate manifests, node/runtime inventory, mount inventory, inside/host mapping record, storage matrix, independent RBAC and capability decision, failure drill, rollback reference and owner approvals.

For engineers building these habits in a structured learning environment, Refonte Learning's DevOps Engineer Program lists a three-month duration, 12–14 hours per week, and a requirement that participants be engaged in bachelor's or postgraduate studies; its published curriculum includes Linux and scripting, Git/GitHub, CI/CD, Docker and Kubernetes, Terraform, AWS/Azure/GCP, monitoring and logging, and a capstone project. Those foundations map naturally to version-controlled manifests, Linux validation and rollout evidence, but the live page does not verify a specific Kubernetes minor version, a user-namespace lab, idmapped-mount exercises or runtime compatibility testing.

Does every Pod use a user namespace once the feature is stable? No. Kubernetes v1.36 marks the feature stable, but a Pod opts in with spec.hostUsers: false. Feature availability and workload adoption are separate decisions. The living documentation also states that the old feature gate is locked/ignored in current releases; that does not rewrite existing Pod specs into user-namespaced Pods.

Will root still appear as UID 0 inside the container? It can. That is the central point of the model: a process can have root privileges inside its user namespace while mapping to an unprivileged host identity outside it. Your proof is the inside/host namespace and mapping evidence, not the word root from id alone.

Do service-account or RBAC permissions change because hostUsers is false? No documented behavior connects user-namespace UID remapping to Kubernetes API authorization. RBAC is evaluated separately. Use application-specific service accounts and authorized kubectl auth can-i checks to prove intended API access.

Does all storage work with user namespaces? No. Kubernetes requires idmapped-mount support for the kubelet Pod-directory filesystem and all Pod-volume filesystems, disallows raw block volumeDevices, and at the cutoff explicitly states that NFS volumes cannot be mounted in user-namespace Pods because the Linux NFS client lacks idmapped-mount support. Test the real storage stack rather than extrapolating from a stateless Pod.

What should we do when a candidate fails? Stop promotion, preserve the exact failure evidence, classify the failing layer, and restore the known-good workload manifest if service or data-path behavior is at risk. Do not weaken admission to make the test pass, do not add unexplained host privileges, and do not recursively rewrite production ownership. The next action is workload-specific: identify the missing mapping, runtime or storage evidence, assign the accountable owner, and retest only after that evidence changes.