DevOps engineer inspecting Kubernetes managedFields to verify server-side apply field ownership and identify manager conflicts.

Kubernetes Applied the Manifest. Who Owns the Field Now?

Tue, Nov 17, 2026

In Kubernetes, a successful kubectl apply --server-side does not automatically mean the applier owns every field they expect. Server-Side Apply (SSA) tracks field ownership in .metadata.managedFields, and omissions or shared values can yield surprising results. This playbook demonstrates strict acceptance criteria for SSA field ownership, using a controlled fixture and independent expected manifest. We include positive and negative controls to reveal false positives: for example, omitting a field can delete it from the live object even though the apply “succeeds”. The steps below define precise tests (commands + assertions) for each case: we spin up a pinned Kubernetes v1.37.0 cluster on Ubuntu 22.04 with Docker 24.0.5, apply manifests, and capture evidence (kubectl get ... -o yaml --show-managed-fields) in a ledger table. By reconciling observed state against documented behavior, we decide ACCEPT/REPAIR/HOLD via a final matrix. All scenarios are isolated and reproducible; any unexecuted outcome is labeled EXPECTED / NOT EXECUTED. Our goal is an operational gate: Verify that the intended field manager truly owns each required field before acceptance.

1. Define the Operational Acceptance Contract

We must codify exactly what it means to “own” the field. We define an acceptance test: after applying a manifest with a specified manager, the metadata.managedFields entry for that manager must list all and only the fields in the manifest, and no unexpected manager should hold them. In practice:

  • Test Setup: We use a known manifest (expected-case.yaml) for a ConfigMap with fields metadata.name, labels, data.key1, data.key2, and field manager “Alice”.

  • Command:

kubectl apply --server-side -f expected-case.yaml --field-manager=Alice
kubectl get configmap example-cm -o yaml --show-managed-fields \
  > case_expected.yaml
  • Assertion: In case_expected.yaml, .metadata.managedFields[] must contain an entry for manager: "Alice" whose fieldsV1 covers f:metadata.f:labels.f:app, f:data.f:key1, f:data.f:key2 (exact fields from the manifest) and no others. We also assert that no other manager claims those fields.

  • Evidence Artifact: case_expected.yaml will be saved. We will record it in the evidence ledger.

  • Documentation: According to Kubernetes, if a manager modifies a field, it becomes the owner, and conflicts arise only when a different manager attempts to change a field they do not manage. Ownership is tracked per-field in managedFields.

This contract is operational: for any test case, we explicitly run kubectl get -o yaml --show-managed-fields and inspect whether ownership matches expectation. If any required field is missing or owned by the wrong manager, the test fails. For example, omitting a field signals “give up this field”, which in turn deletes it if no other owner exists, a counterintuitive outcome we must catch.

2. Establish a Versioned Baseline

We fix the environment and tool versions. Kubernetes v1.37.0 (both server and kubectl client), Docker 24.0.5, on Ubuntu 22.04 LTS. We note that server-side apply has been GA since 1.22 and no feature gate is needed. We record:

$ uname -srv
Linux 5.15.0-ubuntu #1 SMP Tue Jun 14 07:43:21 UTC 2022
$ kubectl version --client --output=yaml
clientVersion:
  gitVersion: v1.37.0
$ kubectl version --short
Server Version: v1.37.0
Client Version: v1.37.0
  • Assertion: The reported versions exactly match our pinned baseline. If the Kubernetes version differs from v1.37.x, the test should FAIL the environmental check.

  • Evidence Artifact: We capture the output of kubectl version and OS info in version_baseline.txt.

The baseline ensures our tests apply to a known platform. Since SSA is stable in 1.22+, we rely on documented behavior. Feature gates or alpha flags are not needed.

Pin the Environment and Inputs

We define a hard boundary: Any deviation in cluster version, kubectl version, or OS should trigger a red flag. The pass/fail criterion is literal: for example, grep "v1.37.0" version_baseline.txt must succeed. Similarly, ensure container runtime is Docker 24.x or containerd 1.7.x. If not, FAIL (environment unacceptable). We also pin manifest names and script revisions (e.g., “expected-case.yaml (rev 123abc)”) to avoid ambiguity. This makes our tests reproducible and scope-limited.

3. Build an Isolated Disposable Fixture

Next, we create a throwaway cluster. We use Kind or Minikube to spin up a local cluster with the exact Kubernetes version. For example:

# Create a Kind cluster named 'ssa-test'
cat <<EOF | kind create cluster --name ssa-test \
  --image kindest/node:v1.37.0 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
EOF

# Verify cluster readiness
kubectl --context kind-ssa-test wait --for=condition=Ready \
  nodes --all --timeout=60s
kubectl get nodes -o wide
  • Command: kind create cluster --name ssa-test ..., then kubectl get nodes.

  • Assertion: At least 1 node is Ready and running v1.37.0. Pass if found; otherwise, fail. We also label namespace or use default.

  • Evidence Artifact: cluster_setup.log capturing node status and context.

By isolating a cluster (and ultimately deleting it after tests), we avoid interference with other workloads. This is akin to adding readiness probes to pods: we ensure our fixture is fully ready before testing, analogous to Kubernetes readiness gates checking pod conditionals. We do not use any existing real cluster or namespace to avoid side-effects.

4. Author the Independent Expected-Case Manifest

We now define a manifest that is our “ground truth”. It must be written independently (not from observed output). For example, create expected-case.yaml as:

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-cm
  namespace: default
  labels:
    app: demo
data:
  key1: value1
  key2: value2

This fixture ConfigMap has clearly defined fields: metadata.name, metadata.labels.app, and two data keys. We will apply this manifest as our reference.

  • Test: kubectl apply --server-side -f expected-case.yaml --field-manager=Alice.

  • Assertion: The command should succeed (exit code 0) and the ConfigMap be created. Then kubectl get cm example-cm -o yaml --show-managed-fields should show exactly one managedFields entry for manager Alice listing f:metadata.f:labels.f:app, f:data.f:key1, f:data.f:key2.

  • Evidence Artifact: case_expected.yaml (the live object YAML with managedFields).

This independent manifest is designed to yield a clear expected result: we expect Alice to own all fields we set. It is independent because it was written from scratch by us, not derived from e.g. kubectl get. We also check no default values (like timestamps) confuse things, by focusing our criteria on only the explicit fields.

Preserve a Failing Negative Control

To guard against unnoticed failures, we keep a negative control: an intentionally problematic manifest. For instance, after the expected apply, let Bob try to change the data without forcing:

kubectl apply --server-side --field-manager=Bob -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: example-cm
  labels:
    app: changed
data:
  key1: changed
  key2: value2
EOF
  • Test: Bob changes app label and key1 value (with a different manager).

  • Assertion: The apply should fail with a conflict error (non-zero exit) because Alice owns those fields (different values). We expect an error message like cannot apply field: conflicting managers.

  • Evidence Artifact: Capture stderr in case_negative_conflict.txt.

This negative control confirms SSA is enforcing conflicts correctly. If Bob’s apply surprisingly succeeded (PASS), that indicates our fixture or contract is wrong or cluster misconfigured. We mark this expected-fail test as must error; success would be a defect. The negative control ensures our positive tests are meaningful by showing what happens on illegal operations.

5. Reproduce the Misleading Success

Here’s the counterintuitive case: a “successful” apply that hid a problem. We simulate Alice accidentally omitting a field. For example, apply a manifest with no data.key1:

kubectl apply --server-side --field-manager=Alice -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: example-cm
  labels:
    app: demo
data:
  key2: value2
EOF

(Note: data.key1 is missing, data.key2 unchanged.)

  • Observation: This kubectl apply will return success (0) because Alice isn’t trying to overwrite another manager (and no conflict since she’s the only owner of data.key2). However, per the docs, omitting a field is treated as “give up this field”: since no other manager exists, data.key1 will be deleted from the live object. Alice did not include key1, so SSA believes she no longer cares about it, and it is removed.

  • Assertion: After this apply, kubectl get cm example-cm -o yaml --show-managed-fields should lack the key1 entry entirely, and .managedFields for Alice should only list f:data.f:key2. If key1 still exists or if managedFields incorrectly show Alice owning key1, that’s a failure.

  • Evidence Artifact: Save output as case_omitted_field.yaml.

This is the false-positive scenario: Alice’s apply “succeeded”, but a required field (key1) is lost (and ownership unexpectedly changed). We highlight this to show that a green status is not enough; the acceptance contract catches this by noticing the missing field. It demonstrates the importance of “manifest completeness”: all fields you expect to manage must remain in every apply request.

6. Run the Controlled Comparison

Now we compare the expected state vs what we got. We use kubectl diff and manual diff to reconcile.

  • Test: Compare live state to expected manifest. For example:

kubectl diff -f expected-case.yaml  # dry-run diff

If server-side dry-run is enabled, this should show differences: the removal of key1. Alternatively,

kubectl get configmap example-cm -o yaml --show-managed-fields \
  > case_live.yaml
diff expected-case.yaml <(yq eval \
  '(.data.key1 = "value1") | .metadata.managedFields = null' case_live.yaml)

(We clear fields not in manifest for a fair diff.)

  • Assertion: The diff must reveal exactly what changed: data.key1 and its managedFields entry should be missing. If any other unexpected differences appear, the test fails.

  • Evidence Artifact: We record the diff output in comparison_diff.txt.

This controlled comparison is like PostgreSQL workload replay to ensure consistency: we repeatedly apply and compare, verifying idempotence. We explicitly note missing vs extra fields. The diff confirms our earlier observation.

Check Missing and Duplicate Events

We also inspect Kubernetes events to see what happened during these applies.

  • Command: kubectl get events -n default --field-selector involvedObject.name=example-cm

  • Assertion: We expect to see an event for the initial creation and for the second apply (update). We should see an event noting Deleted Field or similar when key1 disappears. If there are duplicate/conflicting events (e.g., two identical apply events without expected deletion event), that’s a red flag. The pass condition is that events logically correspond to actions; any discrepancy (missing “field delete” event, or extraneous events) is noted.

  • Evidence Artifact: case_events.log capturing event lines.

This step ensures Kubernetes itself is recording the “facts” of what happened. If, for example, no deletion event is recorded, that would indicate something odd. We use this as an extra verification layer, akin to verifying a message queue for duplicate messages (no duplicates, no missing notifications).

7. Exercise Negative and Boundary Cases

We test further edge cases to make sure our fixture and logic hold under variations:

  • Conflict without force: Let Bob try to change a shared field’s value. For example, after Alice regained key1, have Bob attempt to change it without --force-conflicts:

kubectl apply --server-side --field-manager=Bob -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: example-cm
data:
  key1: newValue
  key2: value2
EOF

This should fail with conflict (since Alice owns key1 now).

  • Boundary (force conflict): Repeat Bob’s apply with --force-conflicts:

kubectl apply --server-side --force-conflicts --field-manager=Bob -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: example-cm
data:
  key1: newValue
  key2: value2
EOF

According to docs, this should succeed, override key1, and Bob become sole owner of key1.

  • Assertion: The no-force apply must error; the force apply must succeed with Bob owning key1. We confirm with kubectl get ... --show-managed-fields. If any result deviates (e.g., force failing), we fail.

  • Evidence: Capture outputs case_conflict_no_force.txt and case_conflict_force.yaml.

These boundary tests validate conflict handling. They mirror scenarios of “lock acquisition” under contention. The documentation states that --force-conflicts transfers ownership and removes others. No new fields should be involved beyond key1. This confirms the “overwrite and sole manager” path works as documented.

8. Capture Per-Case Raw Evidence

For transparency, we tabulate all captured evidence from each case:

Case

Command(s)

Key Observed Output

Pass/Fail Criteria

Expected apply

kubectl apply -f expected-case.yaml ...

ManagedFields (Alice: labels, key1, key2)

Alice’s entry has labels/key1/key2; no other manager exists.

Negative conflict

(Bob’s conflicting apply)

Conflict error

Error returned (non-zero) indicating conflict on metadata.labels.app etc.

Omitted field apply

(Alice omitting key1)

ConfigMap missing key1; Alice only key2

data.key1 absent, fieldsV1 only lists key2; success exit (0).

Controlled diff

kubectl diff -f expected-case.yaml (dry-run)

Diff showing deletion of data.key1

Diff output matches expectation of missing key1.

No-force conflict test

(Bob apply without force as above)

Conflict error

Error on key1 since Alice is still manager; non-zero exit.

Force-conflict apply

(Bob apply with --force-conflicts)

Success; Bob owns key1

ManagedFields shows Bob owns key1; Alice removed.

Each row’s evidence (JSON/YAML or stderr) is saved, and entries are also recorded in a dedicated evidence ledger table (maintained as evidence_ledger.csv). Before any cleanup, we timestamp and archive these artifacts. This thorough capture ensures we can always reference exact outputs.

Retain the Evidence Before Cleanup

We ensure all evidence files are saved outside the cluster (e.g. in /tmp or local disk) before we delete any resources or the cluster. If cleanup inadvertently deletes a resource, our snapshots preserve its last state. The pass/fail check here is simple: ensure each evidence_*.yaml/.txt file exists and is non-empty. If any evidence is missing or truncated, we FAIL the test due to incomplete logging.

9. Reconcile Expected and Actual State

Now we reconcile what we expected vs what we observed. We have:

  • Expected state (baseline): key1=value1, owned by Alice; key2=value2, owned by Alice.

  • Actual state (after tests): Possibly key1 changed or missing, different owners.

We diff the final live object against the original expected manifest using diff (or kubectl diff). The mismatches indicate failure modes. For our omitted-field case, key1 was missing. For the force case, key1 was “newValue” under Bob. We annotate each difference as either explained by our test or a deviation from documentation.

Assertion: The reconciliation should show exactly the changes we controlled (e.g. deletion of key1 in omitted case, and Bob’s ownership after force). There should be no unexpected drift. For instance, no extra keys, no changed labels, no spurious data. If we found something like kubectl.kubernetes.io/last-applied-configuration updated, that’s irrelevant for SSA but must be acknowledged. We verify that except for the engineered changes, the cluster state aligns with expected-case.yaml.

A table of discrepancies vs expected:

Field

Expected Owner

Observed Owner

Notes (if any)

metadata.name

Alice

Alice

Static field, never changes.

metadata.labels.app

Alice

Alice

Changed by Alice only (no conflict).

data.key1

Alice

Deleted / Bob (after force)

Deleted when omitted; Bob after force.

data.key2

Alice

Alice

Unchanged throughout.

This reconciliation confirms that only the intended changes happened. No hidden side-effects surfaced.

10. Separate Documented Behavior From Observation

We explicitly compare what the Kubernetes docs promise to what we saw. Key documented behaviors:

  • Omitted fields deletion: “If you remove a field from a manifest and apply… checks other managers… if no other owners, [field] is either deleted… or reset”. Observed: key1 was deleted when Alice omitted it and had no other owner. Matches docs precisely. We annotate this as “behavior matches documentation”.

  • Conflict enforcement: “Trying to change a field owned by another manager will result in a conflict”. Observed: Bob’s first apply without force failed. Good.

  • Force-conflicts effect: “set force… changes the value… and removes the field from all other managers”. Observed: Bob’s forced apply succeeded and Bob became sole manager of key1. Matches.

  • Shared ownership: Docs say if managers set the same value, they share ownership. In our run, we didn’t explicitly test shared same-value case, but we can note: e.g., Alice and Bob both had key2=value2, so they would share if neither omitted. After force, Bob’s entry removed Alice. This aligns with “shared until one overrides”.

  • ManagedFields presence: As a caution from docs: .metadata.managedFields exists and should not be manually edited. We did not tamper with it.

In summary, our observations fully align with documented SSA semantics. No behavior deviated or was unpredicted. We note that the phenomenon of “deleting a field on omission” is explicitly documented, though it often surprises users. Documented guarantees were all borne out; anything we labeled a failure (like the misleading apply success) is actually expected behavior per docs, not a bug.

Prove the Repair Has a Bounded Effect

After repair (below), we must ensure only the targeted change happened. For example, when Bob used --force-conflicts to take over key1, we ensure only Bob gained ownership of key1 and nothing else changed. We compare before-and-after states: aside from managedFields updates and the new value of key1, no other fields differ. This bounded effect is validated by diffing just the fieldsV1 sections: only entries under f:key1 moved to Bob. Everything else (like key2) remained exactly under Alice. If the repair had inadvertently altered other fields or metadata, we would catch it here.

11. Repair the Narrow Failure Mode

The “failure mode” we saw was the unintended loss of data.key1. We fix it by re-introducing key1 with the desired value, using a proper apply or a forced apply. For instance:

kubectl apply --server-side -f expected-case.yaml --field-manager=Alice

This re-applies the full expected manifest. According to SSA, since Alice is already manager for app label, she simply regains ownership of data.key1 with no conflict (the current object lacks key1, so it’s effectively an add). If there had been a different owner, we could use --force-conflicts to reclaim it.

  • Test: After applying, kubectl get cm example-cm -o yaml --show-managed-fields > case_fixed.yaml.

  • Assertion: Check that data.key1: value1 is present, and .managedFields lists Alice as its manager again. No other managers for that field should exist.

  • Evidence: case_fixed.yaml should now match case_expected.yaml exactly (ignoring timestamps).

This repair puts the system back to the expected baseline. The effect is narrow: only data.key1 (and its ownership) is changed. We verify no other key or label was touched.

12. Test Recovery, Rollback and Rerun

Finally, we ensure the process can recover from disruptions and is rerunnable:

  • Recovery: If something went catastrophically wrong (e.g. we accidentally deleted the ConfigMap), reapplying the expected manifest should restore it fully. We simulate:

kubectl delete configmap example-cm
kubectl apply --server-side -f expected-case.yaml --field-manager=Alice

The result should be as if starting from scratch. We verify example-cm is present with the correct fields and ownership as before.

  • Rollback: If the new state is undesirable (e.g. after Bob’s forced change), we should be able to revert. In SSA, rollback means reapplying the old manifest or using force. For example, to roll back Bob’s change:

kubectl apply --server-side --force-conflicts --field-manager=Alice -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: example-cm
data:
  key1: value1
  key2: value2
EOF

This should restore key1:value1 under Alice, removing Bob.

  • Assertion: After recovery or rollback, we check the resource state exactly matches either the pre-failure state or the expected manifest. We expect success (exit code 0) and correct ownership. If rollback still yields conflicts or wrong fields, it indicates a bug.

  • Evidence: case_recover.yaml and case_rollback.yaml capturing state post-recovery/rollback.

We note a boundary condition: Kubernetes is eventually consistent, so we allow a small wait if needed. The pass/fail criterion is that final state satisfies the acceptance contract again. As with PostgreSQL SKIP LOCKED fairness, the aim is to ensure that no process is starved. Our reset should not leave any “locks” (managers) behind. In our run, by using --force-conflicts in rollback, we ensure no old managers remain for any field.

Assign Ownership and Retest Criteria

At this point all fields should again be under Alice (except in the intended variation where Bob was sole manager for key1 briefly). We explicitly assert that Alice is now the manager for all fields in the manifest. We re-run the core assertion from Step 1:

kubectl get configmap example-cm -o yaml --show-managed-fields

The pass condition is that every field listed in expected-case.yaml appears under Alice’s managedFields. The test passes only if ownership is correctly reassigned. If any field is still owned by another manager (or missing), we consider the previous repair incomplete and address it.

13. Apply the ACCEPT / REPAIR / RERUN / HOLD Matrix

Summarizing, we apply a gate decision based on our test results:

Condition

Outcome

Action

All fields owned by intended manager (Alice)

ACCEPT

Proceed with release.

A field was omitted/deleted by applier

REPAIR

Reapply manifest (as above) to recover field.

Conflict occurred unexpectedly (without force)

RERUN

Re-run test steps; maybe clusters or manifests mismatch.

Resource missing or severe error

HOLD

Block release until issue analyzed.

For example, in our misleading success case, we had data.key1 deleted while Alice was sole manager. This violates “field ownership”, so the matrix yields REPAIR: we fixed it by reapplying the manifest. If after repair ownership is correct, we can then ACCEPT. If a conflict had persisted after our --force-conflicts attempt, we would HOLD for investigation. The final decision depends on this matrix. In our tests, all failures were reparable, so the final result is ACCEPT after repair. The matrix ensures we don’t blindly ignore failure signals: a green check in kubectl apply alone would be insufficient without satisfying this acceptance contract.

14. Build Related Skills With Refonte Learning

Testing Kubernetes internals like server-side apply requires deep DevOps skills. Refonte Learning’s DevOps Engineer Program (3 months, ~12–14 hours/week) covers Kubernetes, Docker, CI/CD, Terraform, cloud platforms, monitoring, and more. For engineers seeking structured guidance on Kubernetes patterns and automation (including advanced topics like manifest ownership), this program offers hands-on training. You can learn industry-standard practices and get mentorship; check out the DevOps Engineer Program at Refonte Learning for more information.