A Kubernetes Job Failed. Which Pod Failure Counts Toward the Retry Limit?

A Kubernetes Job can contain a failed Pod and still be on a valid path to success. It can also contain a failed Pod whose failure immediately consumes retry budget, a failure that is explicitly ignored, a failure that terminates one index without consuming another index's retry budget, or a failure that ends the whole Job regardless of the remaining backoffLimit. Reading kubectl get pods and counting red rows is therefore not a sufficient release decision.

The narrow operational question is: which failed Pod is evidence against which retry limit, and what controller state proves the final outcome? Kubernetes documents that podFailurePolicy can classify failed Pods with Count, Ignore, FailJob, or FailIndex; ordinary Pod failures are otherwise evaluated against backoffLimit, while Indexed Jobs can use backoffLimitPerIndex and expose failedIndexes. Jobs with a Pod failure policy also have constrained replacement semantics: Kubernetes waits for terminal Pod state before evaluating the policy, and modern Job completion conditions distinguish an early failure target from the final Failed condition after Job Pods terminate.

This playbook treats those statements as documented contracts, not lab results. It provides an executable disposable fixture, an independently authored expected-case oracle, evidence commands, a negative control, and an ACCEPT / REPAIR / RERUN / HOLD gate. Because no Kubernetes runtime was available in the authoring environment, every proposed Kubernetes result is labeled EXPECTED / NOT EXECUTED. These outcomes remain expectations until the fixture is run and its raw artifacts are retained.

Define the Operational Contract and Release Gate

The release gate must answer five questions separately: did a Pod reach terminal Failed; which podFailurePolicy rule, if any, matched it; did that failure count against global or per-index backoff; did the controller create a replacement Pod; and what Job condition constitutes final completion evidence? Kubernetes' Job documentation says failed Pods normally count toward .spec.backoffLimit, while a Pod failure policy can override that treatment. The batch/v1 Job API reference defines Ignore as not incrementing the backoff counter, Count as default counting, FailJob as failing the Job, and FailIndex as failing the relevant index.

The acceptance contract is deliberately stronger than “the Job eventually says Complete.” For every case, preserve the submitted manifest, rendered image reference, Job UID, every owned Pod UID, each Pod phase and container exit code, policy rule expected to match, replacement lineage, status.failed, status.succeeded, status.completedIndexes, status.failedIndexes, status.terminating, and Job conditions. On Kubernetes v1.31 and later, terminal Complete or Failed conditions are delayed until Job Pods have terminated; FailureTarget or SuccessCriteriaMet can appear earlier.

A case passes only if the raw API state satisfies the independently written expectation. A green command is not enough. A failed Pod whose policy action is Ignore is evidence of an attempt failure but not, by contract, evidence that retry budget was consumed. Conversely, a FailJob match is sufficient to target the Job for failure without waiting for backoffLimit exhaustion.

Refonte Learning's guide to Kubernetes CronJob missed schedule reconciliation covers scheduled business periods and durable business output, a separate accounting question from Pod-failure retry accounting.

The release verdict vocabulary is fixed: ACCEPT means observed evidence exactly matches the contract; REPAIR means the test or workflow cannot prove the intended property; RERUN means the fixture is valid but execution was interrupted or contaminated; HOLD means evidence is contradictory, missing, or exposes an unexplained version divergence.

Capture the Baseline and Versioned Environment

Do not infer runtime semantics from the article date. Before creating a namespace, capture the client, server, context, nodes, and effective API schema. This playbook was authored against the Kubernetes v1.37 documentation baseline. Kubernetes lists v1.37, v1.36, and v1.35 among maintained release branches, but the relevant evidence is the exact patch version returned by the test cluster.

Execution status: EXPECTED / NOT EXECUTED. The authoring environment was checked on 2026-11-07 and did not contain kubectl, kind, k3d, minikube, docker, or podman; therefore no Kubernetes fixture was run and no Kubernetes outcome below is represented as observed. The host reported Linux kernel 6.18.57 on x86_64. Runtime version fields, source hashes, timestamps, Pod UIDs, container image IDs, command statuses, warnings, and controller observations remain evidence fields to populate during an actual run.

Use a fresh evidence directory and make commands fail visibly:

set -euo pipefail
RUN="$(date -u +%Y%m%dT%H%M%SZ)-$RANDOM"
E="evidence-$RUN"
mkdir -p "$E"

kubectl version -o json >"$E/kubectl-version.json"
kubectl config current-context >"$E/context.txt"
kubectl cluster-info >"$E/cluster-info.txt" 2>"$E/cluster-info.stderr"
kubectl get nodes -o json >"$E/nodes.json"

kubectl explain job.spec.podFailurePolicy --recursive \
  >"$E/explain-pod-failure-policy.txt"
kubectl explain job.spec.backoffLimitPerIndex \
  >"$E/explain-backoff-limit-per-index.txt"
kubectl explain job.spec.podReplacementPolicy \
  >"$E/explain-pod-replacement-policy.txt"

sha256sum "$E"/* >"$E/baseline.sha256"

Record each command's exit status separately if your harness does not already abort on nonzero status. Save stderr rather than discarding warnings. The run is REPAIR before testing if the API server does not expose fields used by the manifest, or if client/server skew violates your organization's supported policy.

The key version boundaries are documented. podFailurePolicy has been stable since Kubernetes v1.31. Per-index backoff is documented as stable from v1.33. Pod replacement policy for Jobs reached GA in v1.34. Kubernetes also documents that, when podFailurePolicy is used, Failed is the allowed replacement policy; the controller evaluates the policy only after the Pod reaches terminal Failed.

Capture the exact container runtime-resolved image IDs after Pods exist:

kubectl get pods -n "$NS" -o json \
  | jq '[.items[] |
      {name:.metadata.name,
       uid:.metadata.uid,
       image:.spec.containers[0].image,
       imageID:.status.containerStatuses[0].imageID}]' \
  >"$E/image-identities.json"

A tag in the source manifest is not equivalent to an immutable runtime identity. For a release-quality fixture, render an approved digest before applying and hash the rendered source.

Construct a Disposable Controlled Fixture

Use one unique namespace, one label identifying the run, no host mounts, no external database, and no production credentials. The fixture requires only the Job controller, ordinary container execution, and kubectl exec for two manually triggered cases.

The independently authored case manifest below defines four Jobs. Replace <BUSYBOX_DIGEST> with a real approved digest before execution.

apiVersion: v1
kind: Namespace
metadata:
  name: job-retry-evidence
---
apiVersion: batch/v1
kind: Job
metadata:
  name: ignore-then-success
  namespace: job-retry-evidence
  labels: {evidence-case: ignore}
spec:
  backoffLimit: 0
  podFailurePolicy:
    rules:
    - action: Ignore
      onExitCodes:
        containerName: main
        operator: In
        values: [42]
  template:
    metadata:
      labels: {evidence-case: ignore}
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 1
      containers:
      - name: main
        image: docker.io/library/busybox@sha256:<BUSYBOX_DIGEST>
        command: ["/bin/sh","-c"]
        args:
        - |
          while :; do
            [ -f /tmp/fail42 ] && exit 42
            [ -f /tmp/succeed ] && exit 0
            sleep 1
          done
---
apiVersion: batch/v1
kind: Job
metadata:
  name: count-negative-control
  namespace: job-retry-evidence
  labels: {evidence-case: count}
spec:
  backoffLimit: 0
  podFailurePolicy:
    rules:
    - action: Count
      onExitCodes:
        containerName: main
        operator: In
        values: [7]
  template:
    metadata:
      labels: {evidence-case: count}
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 1
      containers:
      - name: main
        image: docker.io/library/busybox@sha256:<BUSYBOX_DIGEST>
        command: ["/bin/sh","-c"]
        args:
        - |
          while :; do
            [ -f /tmp/fail7 ] && exit 7
            sleep 1
          done
---
apiVersion: batch/v1
kind: Job
metadata:
  name: failjob-immediate
  namespace: job-retry-evidence
  labels: {evidence-case: failjob}
spec:
  backoffLimit: 6
  podFailurePolicy:
    rules:
    - action: FailJob
      onExitCodes:
        containerName: main
        operator: In
        values: [13]
  template:
    metadata:
      labels: {evidence-case: failjob}
    spec:
      restartPolicy: Never
      containers:
      - name: main
        image: docker.io/library/busybox@sha256:<BUSYBOX_DIGEST>
        command: ["/bin/sh","-c","exit 13"]
---
apiVersion: batch/v1
kind: Job
metadata:
  name: indexed-failindex
  namespace: job-retry-evidence
  labels: {evidence-case: failindex}
spec:
  completionMode: Indexed
  completions: 3
  parallelism: 3
  backoffLimitPerIndex: 0
  maxFailedIndexes: 1
  podFailurePolicy:
    rules:
    - action: FailIndex
      onExitCodes:
        containerName: main
        operator: In
        values: [42]
  template:
    metadata:
      labels: {evidence-case: failindex}
    spec:
      restartPolicy: Never
      containers:
      - name: main
        image: docker.io/library/busybox@sha256:<BUSYBOX_DIGEST>
        command: ["/bin/sh","-c"]
        args:
        - |
          if [ "$JOB_COMPLETION_INDEX" = "0" ]; then
            exit 42
          fi
          exit 0

backoffLimit: 0 is intentional in the negative control: the expected contract is “no retry after a counted failure.” Kubernetes' API describes the field as the number of retries permitted before failure, and Job examples commonly use zero where any Pod failure should fail the Job.

Pin Inputs and Isolate Side Effects

Render, inspect, hash, and then apply:

export NS=job-retry-evidence
cp cases.yaml "$E/cases.rendered.yaml"
sha256sum "$E/cases.rendered.yaml" >"$E/cases.rendered.yaml.sha256"

kubectl apply --server-side --field-manager=job-retry-evidence \
  -f "$E/cases.rendered.yaml" \
  >"$E/apply.stdout" 2>"$E/apply.stderr"
printf '%s\n' "$?" >"$E/apply.status"

A specific boundary assertion is: before triggering the interactive cases, exactly one active Pod exists for ignore-then-success and exactly one for count-negative-control; no Pod from any prior run shares the namespace or case labels.

kubectl get pods -n "$NS" -o json >"$E/pods-before-trigger.json"

If old Pods, Jobs, or foreign labels are present, classify the run RERUN, delete the namespace, and start under a new run identifier. The controlled-fixture principle also applies to Kubernetes StatefulSet scale-down PVC retention validation, whose acceptance target is PVC/PV identity and retained data. This fixture excludes storage and focuses on Pod-failure retry accounting.

Define an Independent Evidence Oracle

Write expected outcomes before looking at runtime status. Otherwise the test risks becoming a description of whatever Kubernetes happened to show.

The independent oracle is:

These expectations follow documented actions rather than a copied Kubernetes tutorial. Ignore explicitly leaves the .backoffLimit counter unincremented and requests a replacement Pod. Count increments the ordinary counter. FailIndex is valid with per-index backoff and marks that index failed without restarting it.

For indexed Jobs, Kubernetes records successful indexes in status.completedIndexes and failed indexes in status.failedIndexes; a failed index does not stop other indexes by itself. Once all indexes finish, the existence of a failed index causes the Job to be marked failed when per-index backoff semantics apply. maxFailedIndexes can terminate the remaining work once its threshold is exceeded.

The oracle must not use status.failed alone as the denominator. The API documentation links that field to counted failures for ordinary policy handling, but Ignore, FailJob, and FailIndex change the interpretation. The actual proof is the tuple of Pod phase, exit status, policy rule, index identity where relevant, replacement identity, and Job conditions.

Reproduce the Misleading Success Case

The counterintuitive case is a Job that ends Complete even though one owned Pod previously reached Failed. That is expected when the failure matches an Ignore rule and the replacement later succeeds.

First capture the original Pod identity:

P1="$(kubectl get pod -n "$NS" -l job-name=ignore-then-success \
  -o jsonpath='{.items[0].metadata.name}')"
U1="$(kubectl get pod -n "$NS" "$P1" \
  -o jsonpath='{.metadata.uid}')"

kubectl get pod -n "$NS" "$P1" -o json >"$E/ignore-pod-before.json"
kubectl exec -n "$NS" "$P1" -- touch /tmp/fail42

Wait for that Pod to become terminal, preserving it before any cleanup:

kubectl wait -n "$NS" --for=jsonpath='{.status.phase}'=Failed \
  "pod/$P1" --timeout=60s

kubectl get pod -n "$NS" "$P1" -o json >"$E/ignore-failed-pod.json"
kubectl get job -n "$NS" ignore-then-success -o json \
  >"$E/ignore-job-after-failure.json"

Kubernetes documents that a Job using podFailurePolicy matches only Pods in terminal Failed, and that such Jobs use the delayed replacement behavior associated with podReplacementPolicy: Failed. In current semantics, the replacement should therefore be a new Pod object created after the failed attempt reaches terminal state, not merely the same container restarted inside the old Pod.

Find the new UID, require it to differ from U1, and trigger success:

P2="$(
  kubectl get pod -n "$NS" -l job-name=ignore-then-success -o json |
  jq -r --arg u "$U1" '.items[] | select(.metadata.uid != $u) | .metadata.name' |
  head -n1
)"
test -n "$P2"

kubectl exec -n "$NS" "$P2" -- touch /tmp/succeed
kubectl wait -n "$NS" --for=condition=complete \
  job/ignore-then-success --timeout=60s

EXPECTED / NOT EXECUTED: one historical Pod has phase Failed and exit code 42; a different Pod UID succeeds; the Job becomes Complete; the ignored failure must not be interpreted as having consumed the zero retry allowance. That is the answer to the title question in its most compact form: a failed Pod can exist without counting toward the retry limit when a matching podFailurePolicy rule says Ignore.

Record the Negative Control Before Repair

Do not run only the friendly case. Trigger the otherwise identical negative control with exit 7:

N1="$(kubectl get pod -n "$NS" -l job-name=count-negative-control \
  -o jsonpath='{.items[0].metadata.name}')"
kubectl exec -n "$NS" "$N1" -- touch /tmp/fail7

kubectl wait -n "$NS" --for=condition=failed \
  job/count-negative-control --timeout=60s || true

kubectl get pod -n "$NS" -l job-name=count-negative-control -o json \
  >"$E/count-pods.json"
kubectl get job -n "$NS" count-negative-control -o json \
  >"$E/count-job.json"

The measurable assertion is: the Count case must not acquire a successful replacement path under backoffLimit: 0, while the Ignore case must. If both behave identically, the fixture has failed to discriminate the policy and receives REPAIR, not ACCEPT.

Run a Contrasting Correctness Case

The failjob-immediate case distinguishes “this failure counts toward a retry budget” from “this failure bypasses that budget as a terminal policy decision.” Its Pod exits 13 immediately, while backoffLimit is deliberately six.

kubectl get job -n "$NS" failjob-immediate -o json \
  >"$E/failjob-job.json"
kubectl get pods -n "$NS" -l job-name=failjob-immediate -o json \
  >"$E/failjob-pods.json"

The expected contract is not “six retries occurred quickly.” It is that exit 13 matches FailJob, causing the controller to target the Job for failure. Kubernetes' task documentation says a matching FailJob rule causes the controller to add FailureTarget, and the terminal Failed condition appears after all Job Pods terminate.

Inspect conditions explicitly:

jq '.status.conditions //
    [] | map({type,status,reason,message,lastTransitionTime})' \
  "$E/failjob-job.json"

Do not collapse FailureTarget and Failed. On current Kubernetes, they answer different timing questions. FailureTarget=True means the controller has decided the Job meets failure criteria and is terminating remaining work. Failed=True is the terminal completion evidence after its Pods have terminated.

A measurable pass condition is: no evidence suggests backoffLimit exhaustion was necessary to reach the policy-driven failure decision. If the Job's terminal reason indicates something unrelated, such as an active deadline, classify the case HOLD until the interference is explained.

Explore Failure and Boundary Conditions

The indexed case answers a separate accounting question: when retries are scoped per index, which work unit is considered failed?

Kubernetes allows backoffLimitPerIndex only for Indexed Jobs with restartPolicy: Never. It records per-index failures and successful indexes separately, and FailIndex can mark one index failed without retrying it.

The supplied fixture has three indexes. Index 0 exits 42. Indexes 1 and 2 exit zero. Collect both Pod annotations and Job status:

kubectl wait -n "$NS" --for=condition=failed \
  job/indexed-failindex --timeout=120s || true

kubectl get pods -n "$NS" -l job-name=indexed-failindex -o json \
  >"$E/indexed-pods.json"
kubectl get job -n "$NS" indexed-failindex -o json \
  >"$E/indexed-job.json"

jq '[.items[] | {
  name:.metadata.name,
  uid:.metadata.uid,
  phase:.status.phase,
  index:.metadata.annotations["batch.kubernetes.io/job-completion-index"],
  failureCount:
    .metadata.annotations["batch.kubernetes.io/job-index-failure-count"],
  exitCode:.status.containerStatuses[0].state.terminated.exitCode
}]' "$E/indexed-pods.json"

EXPECTED / NOT EXECUTED: index 0 appears in status.failedIndexes; indexes 1 and 2 appear in status.completedIndexes; index 0 is not retried because FailIndex is the matching action. The overall Job should eventually be terminal Failed, even though two-thirds of the indexes succeeded. Kubernetes explicitly documents that an Indexed Job using per-index backoff is marked failed if any index failed after the remaining indexes finish, unless other configured semantics alter the termination path.

maxFailedIndexes: 1 does not mean the Job fails as soon as the first failed index exists; the documented early-termination threshold is exceeded when the number of failed indexes goes beyond the configured maximum. Keep that threshold arithmetic separate from the FailIndex decision itself.

Check Missing and Duplicate Evidence

A boundary failure is not only “wrong status.” Missing evidence can make a technically plausible result unacceptable.

Assert:

test "$(jq '[.items[].metadata.uid] | unique | length' \
  "$E/indexed-pods.json")" \
  = "$(jq '.items | length' "$E/indexed-pods.json")"

Then reconcile completion indexes from two independent places: the Job's compressed status strings and each Pod's completion-index annotation. If completedIndexes says 1,2 but no retained successful Pod or other preserved artifact maps to index 2, classify HOLD. If two terminal successful Pods claim the same index and the evidence window cannot explain why, classify HOLD rather than deciding from names alone.

The measurable assertion is identity-based: every expected index has exactly one terminal disposition in the ledger, either completed or failed, and no index is both.

Capture Complete Per-Case Evidence

The evidence bundle should be reproducible without rerunning kubectl get after objects have changed. Capture raw JSON first, then derive human-readable summaries.

for job in \
  ignore-then-success \
  count-negative-control \
  failjob-immediate \
  indexed-failindex
do
  kubectl get job -n "$NS" "$job" -o json \
    >"$E/$job.job.json"
  kubectl get pods -n "$NS" -l "job-name=$job" -o json \
    >"$E/$job.pods.json"
  kubectl describe job -n "$NS" "$job" \
    >"$E/$job.describe.txt"
done

kubectl get events -n "$NS" \
  --sort-by=.metadata.creationTimestamp -o json \
  >"$E/events.json"

sha256sum "$E"/* >"$E/final-artifacts.sha256"

Record the ledger in a machine-readable file as well as prose:

The ledger must include kubectl status codes and stderr from mutating commands. A command such as kubectl exec returning zero proves only that the API request and exec session succeeded; it does not prove the container subsequently terminated with the intended exit code. That proof comes from the Pod's terminated container status.

Likewise, kubectl wait --for=condition=complete is a convenience synchronizer, not an independent oracle. The raw Job and Pod JSON remain the acceptance artifacts.

Reconcile Expected and Observed Identities

Names are insufficient for replacement analysis. A Job replacement Pod can share labels and operational purpose with its predecessor while having a different immutable Kubernetes UID. Record both.

For each Job, emit a chronological identity table:

jq -r '
  .items
  | sort_by(.metadata.creationTimestamp)
  | .[]
  | [
      .metadata.creationTimestamp,
      .metadata.name,
      .metadata.uid,
      (.metadata.annotations["batch.kubernetes.io/job-completion-index"] // "-"),
      .status.phase,
      (.status.containerStatuses[0].state.terminated.exitCode // "-")
    ]
  | @tsv
' "$E/ignore-then-success.pods.json" \
  >"$E/ignore.identity.tsv"

For the Ignore case, the expected identity chain is UID-A: Failed/42 -> UID-B: Succeeded/0. The claim being tested is not merely “two Pod names existed.” It is that the controller accepted a terminal failed attempt under an Ignore rule, did not charge that attempt to the zero retry allowance, then created a new Pod that supplied completion evidence. That interpretation comes directly from the API definition of Ignore.

For the Count negative control, there should not be a legitimate success chain after the counted failure when the allowed retry count is zero. For FailIndex, the relevant identity is (Job UID, completion index, Pod UID) because the index, rather than the whole Job, is the retry-accounting unit.

The method resembles the identity discipline in Refonte Learning's Ansible check mode task coverage validation: that article distinguishes invocation, execution, and actual validation; here the analogous distinction is Pod existence, policy accounting, and Job completion evidence.

Retain Failure Artifacts Before Rerun

Before deleting or rerunning anything, require:

test -s "$E/final-artifacts.sha256"
test -s "$E/events.json"
test -s "$E/ignore-then-success.job.json"
test -s "$E/ignore-then-success.pods.json"
test -s "$E/count-negative-control.job.json"
test -s "$E/indexed-failindex.job.json"

A boundary example: if the Count case unexpectedly completes, do not rerun immediately. Preserve the Job UID, Pod UIDs, rendered manifest hash, events, versions, and container image IDs first. The measurable assertion is that the evidence directory remains immutable after failure triage begins; a later green run must receive a new run ID rather than overwrite the failed run.

Separate Documented Behavior From Observation

A defensible Kubernetes validation record uses four statement classes.

Documented contract: Kubernetes controllers are reconciliation loops that drive actual cluster state toward desired state through the API. The Job controller creates or removes Pods and updates Job status; it does not execute application code itself.

Documented contract: A Job's Pod failure policy is evaluated for failed Pods, rules are processed in order, and the first matching rule determines handling. If no rule matches, default counting applies. Ignore, Count, FailJob, and FailIndex have distinct API meanings.

Documented contract: Kubernetes v1.27 and later normally transitions deleted Pods to a terminal phase before API deletion, with documented exceptions. This matters because Job failure policy matching depends on terminal state.

Documented contract: In Kubernetes v1.31 and later, terminal Job conditions are delayed until Job Pods terminate. FailureTarget and SuccessCriteriaMet expose the earlier controller decision point.

Test hypothesis: A failed Pod exiting 42 under our Ignore rule will be retained as failure evidence, not consume the zero backoff allowance, and be followed by a replacement that can complete the Job.

Test hypothesis: Exit 7 under Count with backoffLimit: 0 will fail the negative-control Job without a permitted retry.

Test hypothesis: Exit 13 will match FailJob, making retry exhaustion irrelevant to the failure decision.

Test hypothesis: Index 0 will enter failedIndexes under FailIndex, while indexes 1 and 2 complete.

Actual observation: None of the four Kubernetes hypotheses was executed.

That separation matters because documentation applicability is versioned. For example, the historical v1.28 Job API introduced replacement-policy behavior before it later became GA; copying an old blog's feature-state wording into a v1.37 run would be inaccurate even if the core behavior remained related.

Repair the Workflow Without Masking Errors

Repair means improving evidence quality or correcting a manifest, not changing the test until it becomes green.

If the Ignore case produces no replacement, first check the failed Pod's actual exit code and terminal phase. A policy rule cannot match a code that was never produced. Check the effective Job spec returned by the API rather than the local YAML. Check admission mutations and managedFields. Confirm that no deadline, deletion, or unrelated controller action terminated the Job first.

If the negative control unexpectedly creates a replacement, check the exact server version and whether the failure really matched the Count rule. Preserve status.failed, conditions, event stream, and all Pod UIDs before any change. Do not increase backoffLimit merely to make the test finish differently; that would alter the contract being validated.

If the indexed case lacks failedIndexes, confirm completionMode: Indexed, backoffLimitPerIndex, and FailIndex survived admission. The API requires FailIndex to be used with per-index backoff.

A repair may also target the collector. For example, this is unsafe:

kubectl get pod -l job-name=ignore-then-success \
  -o jsonpath='{.items[0].metadata.name}'

after replacement has occurred, because array order does not prove which Pod is historical. The repaired collector sorts by creation timestamp and stores UIDs before triggering the failure.

Explain the Scope of the Fix

A concrete boundary: changing the evidence collector from “first Pod in list” to “the UID recorded before the trigger” repairs identity attribution only. It does not change Kubernetes retry behavior.

The measurable assertion is that each repair has one named scope:

A repair is accepted only when the original failed evidence remains preserved and the rerun receives a new run identifier.

Test Recovery, Rollback and Rerun Safety

The fixture has no external business writes, so rollback is namespace deletion after evidence retention:

kubectl delete namespace "$NS" --wait=true \
  >"$E/cleanup.stdout" 2>"$E/cleanup.stderr"
printf '%s\n' "$?" >"$E/cleanup.status"

Do not delete the namespace before final Job and Pod JSON are captured. Kubernetes object garbage collection is operational cleanup, not evidence retention.

Rerun safety requires a new namespace or run label and a freshly rendered manifest hash. Never reuse old Pod UIDs as expected values. The controller may legitimately generate different names, UIDs, timestamps, scheduling nodes, and resource versions while still satisfying the same semantic contract.

A valid rerun also re-captures the exact Kubernetes version. Patch-level differences can matter to controller bugs even when the API contract is nominally unchanged. This is why the evidence directory contains kubectl-version.json rather than a prose statement such as “tested on Kubernetes 1.37.”

The recovery principle parallels Refonte Learning's Terraform moved block state refactor: that article retains identity evidence before accepting a state migration; this Job playbook instead retains attempt and controller identities before retrying a failed validation. The acceptance targets and fixtures are different.

For rollback safety, also capture whether cleanup itself succeeded. A nonzero namespace-delete command is not grounds to rewrite the experiment as failed; classify the validation result and cleanup result independently. If lingering objects could contaminate the next run, cleanup failure converts the next run to HOLD until isolation is restored.

Apply the Acceptance and Ownership Matrix

Use the matrix only after raw evidence has been frozen.

Ownership matters because not all mismatches are application defects. A wrong exit code belongs to the fixture or workload owner. A field rejected by the API is a version/applicability issue. An unexplained controller-status contradiction belongs with the platform or Kubernetes investigation path. A missing artifact caused by the evidence harness is a test-system defect.

The central retry-accounting decision can be reduced to this sequence:

1. Identify the failed Pod by UID, not only by name.

2. Prove terminal Failed and capture the terminated container exit code or matching Pod condition.

3. Read the effective Job's ordered podFailurePolicy.

4. If a rule matches, apply its documented action: Ignore, Count, FailJob, or FailIndex.

5. If no rule matches, default failure counting applies.

6. For Indexed Jobs with per-index backoff, reconcile the completion index and failedIndexes.

7. Prove replacement behavior using a new Pod UID.

8. Treat FailureTarget as an early failure decision and terminal Failed as final failure completion evidence on current Kubernetes.

9. Compare the resulting record to the prewritten oracle.

Record ACCEPT, REPAIR, RERUN or HOLD

The only justified overall verdict is HOLD: fixture NOT EXECUTED. The fixture and oracle are authored, and the Kubernetes contracts were checked against primary documentation, but no Kubernetes server was available to produce runtime evidence.

That does not mean the expected outcomes are doubtful. It means expectation and observation are different evidence classes.

A future executed run can become ACCEPT only if every required field is populated, including exact server version, rendered source hash, actual image ID, command status, Pod UID lineage, exit codes, Job conditions, failed/completed indexes, and cleanup status. A single unexplained conflict (such as an ignored Pod apparently incrementing the retry counter, or a FailIndex Pod being retried under an effective spec that should prevent it) requires HOLD.

Build Related Skills With Refonte Learning

The Refonte Learning DevOps Engineer Program page lists a three-month period, a stated dedication of 12–14 hours per week, Docker and Kubernetes under “Containerization,” Terraform, cloud platforms, monitoring/logging, and a capstone project. The page also states a bachelor's-or-higher study requirement and describes Training and Internship certificates on successful completion. Those statements are page claims, not independent assessments of outcomes.

The page does not establish that this exact Kubernetes Job retry-accounting lab, podFailurePolicy, per-index retry semantics, or a specialized Kubernetes release-acceptance module is taught. These details cannot be inferred from the broader “Docker & Kubernetes” curriculum label. Claims about guaranteed internship placement, mentorship scope for this specific subject, or employment outcomes require confirmation against the current program terms.

For engineers who want structured practice across CI/CD, containerization, infrastructure as code, cloud platforms, and monitoring, review the Refonte Learning DevOps Engineer Program page and compare its current syllabus with the evidence discipline used here.

The Kubernetes-specific acceptance rule remains independent of any course: a failed Pod counts toward the retry limit only according to the effective Job failure policy and retry scope. Count or default handling consumes the relevant backoff budget; Ignore does not; FailJob makes retry exhaustion irrelevant by failing the Job; and FailIndex terminates the relevant indexed work unit when used with per-index backoff. Replacement identity and final Job conditions then supply the evidence needed to decide ACCEPT, REPAIR, RERUN, or HOLD.