The HPA Reported a Replica Target. Which Metrics Were Missing?

An HPA status that contains a current metric and a replica recommendation is not, by itself, evidence that every Pod contributed a metric to that recommendation. Kubernetes deliberately sets missing Pod metrics aside, treats certain CPU metrics from not-yet-ready Pods specially, and then recomputes the ratio conservatively. For scale-down, missing Pods are assumed to consume 100% of the target; for scale-up, they are assumed to consume 0%. Not-yet-ready Pods can also be treated as consuming 0% when the initial result points upward. If that recomputation reverses direction or falls inside tolerance, no scale occurs. Crucially, Kubernetes documents that the HPA status still reports the original average utilization rather than the conservative recomputation.

That creates a narrow but important acceptance problem: a healthy API response, ScalingActive=True, or a plausible desiredReplicas value can coexist with incomplete metric participation. The correct release question is therefore not “Did HPA produce a number?” It is “Which Pods were eligible, which PodMetrics were actually present, which were set aside as missing or not-yet-ready, and does an independent oracle predict the same conservative decision?”

This playbook defines that proof. The fixture below is intentionally disposable, synthetic, version-pinned, and isolated from production. No case in this article was executed for publication. Every proposed result is therefore labeled EXPECTED / NOT EXECUTED, and the release decision is HOLD until the evidence ledger is populated from a real run.

Kubernetes’ resource metrics pipeline is intentionally minimal: metrics-server or another implementation of the Metrics API supplies CPU and memory data that HPA consumes, and kubectl top reads from the same resource-metrics API. CPU utilization additionally depends on Pod resource requests, because utilization is the usage-to-request ratio used by HPA.

Define the acceptance contract

Treat this as release acceptance for one algorithmic boundary, not as a generic autoscaling tutorial. The contract is limited to Kubernetes HPA resource CPU metrics, missing Pod metrics, not-yet-ready Pods, conservative recomputation, and the resulting scale decision.

The first distinction to preserve is between metric transport success and metric completeness. An API request to metrics.k8s.io can succeed while returning metrics for only a subset of Pods. Kubernetes’ HPA algorithm explicitly anticipates that situation: missing Pod metrics are separated from the initial calculation and then introduced into a conservative second pass.

The second distinction is between the reported current utilization and the effective utilization used after conservative substitution. Kubernetes documents that HPA status reports the original average utilization, not the recomputed value that includes assumptions for missing or not-yet-ready Pods. That is the false-positive success signal this playbook is designed to defeat.

Define these acceptance assertions before running anything:

All test cases in this article are EXPECTED / NOT EXECUTED.

This differs materially from Refonte Learning’s Machine learning evaluation, which is about baselines, dataset splits, decision metrics, calibration, uncertainty, and protected test sets rather than Kubernetes controller reconciliation. The shared principle is useful (an output value is not sufficient evidence), but the operational workflow here is different: Kubernetes object identity, Pod readiness, per-Pod metric membership, controller state, and scale mutation are the evidence boundaries.

A useful acceptance artifact is a plain-text contract committed next to the fixture:

CASE CONTRACT
scope: HPA CPU Resource metric only
target: averageUtilization=50
current replicas: case-defined
request per pod: 100m CPU
oracle input: explicit pod/readiness/metric tuples
oracle must not read HPA status.desiredReplicas
evidence must include raw Pod list + raw metrics.k8s.io response
unexecuted status: EXPECTED / NOT EXECUTED
release states: ACCEPT | REPAIR | RERUN | HOLD

The default decision today is HOLD, because specification without observation is not release evidence.

Capture a pinned baseline

A reproducible result starts with versions that can be named again. As of November 12, 2026, Kubernetes maintains the 1.37, 1.36, and 1.35 release branches; the official release page lists 1.37.1 as the latest 1.37 patch published September 15, 2026. For this fixture, use kind v0.33.0 with the Kubernetes v1.37.0 node image that the kind release itself publishes by content digest. kind v0.33.0 defaults to Kubernetes 1.37.0 and publishes the exact image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5.

Do not silently upgrade the fixture to 1.37.1 or a later patch simply because it is newer. The purpose of the pin is not “latest”; it is to make the acceptance result attributable to an exact controller implementation. Kubernetes’ v1.37.0 source contains the replica-calculator implementation under pkg/controller/podautoscaler/replica_calculator.go, and its test suite includes controller scenarios for Pod readiness, metrics and desired replicas.

Baseline commands:

kind version
kubectl version --client -o yaml
docker version
uname -a

kind create cluster \
  --name hpa-missing-metrics \
  --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5

kubectl version -o yaml
kubectl get nodes -o wide
kubectl get --raw /version

Artifact every command rather than copying selected lines:

mkdir -p evidence/baseline

kind version > evidence/baseline/kind-version.txt 2>&1
kubectl version -o yaml > evidence/baseline/kubectl-version.yaml 2>&1
docker version > evidence/baseline/docker-version.txt 2>&1
kubectl get --raw /version > evidence/baseline/apiserver-version.json 2>&1
kubectl get nodes -o yaml > evidence/baseline/nodes.yaml 2>&1

The fixture assumptions are deliberately narrow: cluster provider has no specific constraint; host OS has no specific constraint beyond being able to run the pinned kind image and container runtime; CPU architecture may be amd64 or arm64 because the kind release publishes those node images for both supported platforms. The host container engine version must still be captured because it is part of rerun provenance.

Pin fixture versions

Pass condition: the evidence ledger contains the exact kind version, exact Kubernetes node-image digest, API-server version, kubectl client version, and container-engine version before any case is run.

Fail condition: any runtime is represented only as latest, an unpinned floating image tag is substituted for the node image, or the cluster version differs from the manifest’s declared behavior version.

Status: EXPECTED / NOT EXECUTED.

A practical versions.env:

KIND_VERSION=v0.33.0
K8S_VERSION=v1.37.0
KIND_NODE_IMAGE='kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5'
FIXTURE_SCHEMA=1

Kubernetes documentation also exposes controller-level parameters that matter to the test: the initial readiness delay defaults to 30 seconds, the CPU initialization period defaults to five minutes, and the default tolerance is 10% unless configured otherwise. Capture the effective controller-manager arguments if the fixture changes any of those defaults.

Create an isolated fixture

The fixture needs one namespace, one HPA, one synthetic scalable target, and a metrics source whose PodMetric membership can be controlled. Using a live application workload would weaken the experiment because real CPU sampling, startup timing and scheduler placement would introduce variables unrelated to the missing-metric decision.

A StatefulSet is useful because names are predictable: fixture-0, fixture-1, fixture-2, and so on. Each container declares a 100m CPU request; that makes a synthetic 80m usage equal 80% utilization and keeps the oracle arithmetic visible. Kubernetes requires resource requests for resource-utilization scaling because utilization is evaluated against requested resources.

The target:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: fixture
  namespace: hpa-proof
spec:
  serviceName: fixture
  podManagementPolicy: Parallel
  replicas: 4
  selector:
    matchLabels:
      app: hpa-proof
  template:
    metadata:
      labels:
        app: hpa-proof
    spec:
      containers:
      - name: app
        image: registry.k8s.io/pause:3.10
        resources:
          requests:
            cpu: 100m
            memory: 16Mi
          limits:
            cpu: 100m
            memory: 16Mi

The HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: fixture
  namespace: hpa-proof
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: fixture
  minReplicas: 1
  maxReplicas: 10
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
    scaleDown:
      stabilizationWindowSeconds: 0
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

HPA behavior rules are applied after metric-derived desired replicas and can independently limit or stabilize scaling, so the fixture sets explicit zero stabilization windows to prevent historical recommendations from contaminating the intended comparison. This does not disable tolerance; the manifest must record the effective tolerance separately.

A production metrics-server is intentionally not enough for deterministic missing-metric injection. The Kubernetes resource-metrics pipeline normally has metrics-server collecting CPU and memory from kubelets and serving metrics.k8s.io; that is exactly what HPA consumes. For this fixture, replace that provider with a test-only aggregated Metrics API implementation that returns explicit PodMetrics payloads. Do not install both.

That synthetic provider must be isolated to this disposable cluster. Its API should return only metrics.k8s.io/v1beta1, which remains the HPA resource-metrics API described by Kubernetes documentation. Store each response body as a versioned case file rather than generating values implicitly.

This fixture boundary is different from Refonte Learning’s asyncio context capture article, which tests when Python execution context is captured around coroutine and Task boundaries. Here, timing matters only insofar as Kubernetes readiness classification and metric timestamps affect HPA eligibility; there is no coroutine-context workflow to duplicate.

Commands:

kubectl create namespace hpa-proof
kubectl apply -f fixture/statefulset.yaml
kubectl apply -f fixture/hpa.yaml

kubectl get sts,pods,hpa -n hpa-proof -o wide
kubectl get hpa fixture -n hpa-proof -o yaml \
  > evidence/baseline/hpa-initial.yaml
kubectl get pods -n hpa-proof -l app=hpa-proof -o json \
  > evidence/baseline/pods-initial.json

Status: EXPECTED / NOT EXECUTED.

Write the expected-case manifest

Do not calculate “expected” behavior by reading HPA output after the run. That merely turns the system under test into its own oracle.

The independent expected-case manifest should contain enough information to reproduce Kubernetes’ documented two-pass reasoning without querying status.currentMetrics or status.desiredReplicas. Kubernetes’ documented algorithm first calculates with eligible metric-bearing Pods, sets missing metrics aside, then substitutes conservative values: 100% of target during a possible scale-down and 0% during a possible scale-up. It also treats not-yet-ready Pods as 0% when the first calculation would scale upward.

A compact oracle manifest:

schema: hpa-proof/v1
kubernetesBehaviorVersion: v1.37.0
targetUtilizationPercent: 50
tolerance: 0.10
cases:
  complete-up:
    currentReplicas: 4
    pods:
      fixture-0: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-1: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-2: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-3: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
    expected:
      reportedUtilizationPercent: 80
      conservativeUtilizationPercent: 80
      desiredReplicas: 7
      state: EXPECTED / NOT EXECUTED

  one-missing-up:
    currentReplicas: 4
    pods:
      fixture-0: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-1: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-2: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-3: {ready: true, requestMilliCPU: 100, usageMilliCPU: null}
    expected:
      reportedUtilizationPercent: 80
      conservativeUtilizationPercent: 60
      desiredReplicas: 5
      state: EXPECTED / NOT EXECUTED

  two-missing-reversal:
    currentReplicas: 4
    pods:
      fixture-0: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-1: {ready: true, requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-2: {ready: true, requestMilliCPU: 100, usageMilliCPU: null}
      fixture-3: {ready: true, requestMilliCPU: 100, usageMilliCPU: null}
    expected:
      reportedUtilizationPercent: 80
      conservativeUtilizationPercent: 40
      desiredReplicas: 4
      rationale: conservative recomputation reverses initial scale-up direction
      state: EXPECTED / NOT EXECUTED

  one-not-ready-up:
    currentReplicas: 4
    pods:
      fixture-0: {ready: true,  requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-1: {ready: true,  requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-2: {ready: true,  requestMilliCPU: 100, usageMilliCPU: 80}
      fixture-3: {ready: false, requestMilliCPU: 100, usageMilliCPU: 80}
    expected:
      reportedUtilizationPercent: 80
      conservativeUtilizationPercent: 60
      desiredReplicas: 5
      state: EXPECTED / NOT EXECUTED

For complete-up, 80% / 50% = 1.6; ceil(4 × 1.6) = 7. For one missing metric during an upward recommendation, conservative substitution adds one Pod at 0% target-equivalent utilization, producing (80 + 80 + 80 + 0) / 4 = 60%; 60/50 = 1.2; ceil(4 × 1.2) = 5. Those arithmetic values are independent expected outputs derived from the documented rule, not observed HPA results.

The Kubernetes v1.37.0 source is a secondary implementation oracle: the pinned replica_calculator.go is the implementation being tested, while its test suite demonstrates Kubernetes itself exercises readiness and metric-participation scenarios. Keep your manifest independent anyway; copying controller code would only reproduce the same defect twice.

Establish a negative control

A good negative control proves that the experiment can detect the absence of conservative substitution.

Use the complete four-Pod case as the negative control: every selected Ready Pod receives 80m. The metric membership set and workload set must therefore be identical.

jq -r '.items[].metadata.name' evidence/cases/complete-up/pods.json | sort \
  > /tmp/pods

jq -r '.items[].metadata.name' evidence/cases/complete-up/podmetrics.json | sort \
  > /tmp/metrics

diff -u /tmp/pods /tmp/metrics

Pass: diff is empty, reported utilization is expected to be 80%, and the independent manifest predicts seven replicas.

Fail: any PodMetric is absent, readiness differs from the declared case, or HPA is compared against the missing-metric oracle rather than the complete-case oracle.

Status: EXPECTED / NOT EXECUTED.

Reproduce the misleading success

The interesting case is not a failed Metrics API call. It is a successful evaluation whose visible summary can tempt an operator into assuming complete input.

For one-missing-up, return metrics for three Ready Pods at 80m and omit fixture-3 entirely. The Metrics API request should still return a valid PodMetricsList. HPA’s first average is therefore based on the three available metrics. Kubernetes documents that missing metrics are set aside and later folded back through conservative assumptions.

The evidence capture must not rely on kubectl top alone. kubectl top is a client of the same resource-metrics pipeline, useful for diagnostics but not an independent source. Capture the raw API response:

CASE=one-missing-up
DIR="evidence/cases/$CASE"
mkdir -p "$DIR"

kubectl get pods -n hpa-proof -l app=hpa-proof -o json \
  > "$DIR/pods.json"

kubectl get --raw \
  '/apis/metrics.k8s.io/v1beta1/namespaces/hpa-proof/pods?labelSelector=app%3Dhpa-proof' \
  > "$DIR/podmetrics.json"

kubectl get hpa fixture -n hpa-proof -o json \
  > "$DIR/hpa.json"

kubectl get statefulset fixture -n hpa-proof -o json \
  > "$DIR/statefulset.json"

kubectl get events -n hpa-proof --sort-by=.lastTimestamp -o json \
  > "$DIR/events.json"

Then calculate set membership explicitly:

jq -r '.items[].metadata.name' "$DIR/pods.json" | sort > "$DIR/pod-names.txt"
jq -r '.items[].metadata.name' "$DIR/podmetrics.json" | sort > "$DIR/metric-pod-names.txt"

comm -23 "$DIR/pod-names.txt" "$DIR/metric-pod-names.txt" \
  > "$DIR/missing-metrics.txt"

The expected file contains:

fixture-3

EXPECTED / NOT EXECUTED.

The misleading status pattern is this: HPA may report a current utilization corresponding to the original metric-bearing sample while making its scale decision using the conservative recomputation. Kubernetes explicitly documents that status reports the original average utilization rather than the adjusted average. Therefore, observing “80% / 50%” alongside a smaller-than-naïve replica recommendation is not automatically contradictory.

Do not label ScalingActive=True as “all metrics present.” Kubernetes’ HPA documentation describes that condition as indicating that HPA was able to calculate a desired scale from a valid metric; it does not define it as proof that every Pod contributed a metric.

The acceptance assertion is narrower: successful metric calculation must coexist with an independently proven missing-set of exactly one Pod, and the scale result must equal the conservative oracle rather than the complete-metrics oracle.

Run the comparison case

Every missing-metric result needs an adjacent complete case under the same HPA target, replica count, resource requests, and synthetic usage. Otherwise a replica difference could be blamed on configuration drift.

Set four Ready Pods and return 80m for all four. The expected calculation is 80% current utilization, ratio 1.6, desired replicas seven. Then switch only the metric membership so that fixture-3 disappears; the expected conservative result becomes five.

This comparison targets metric eligibility only. It should not inherit lifecycle concerns from Refonte Learning’s Kubernetes preStop article. That article focuses on Pod termination, preStop execution, termination grace-period consumption, endpoint transition, signal delivery and forcible shutdown. Here, deleting Pods during a measurement window would actually damage the fixture because deleting Pods are excluded from HPA’s per-Pod resource-metric calculation. Kubernetes documents that Pods with deletion timestamps are ignored and failed Pods are discarded.

Therefore, case transitions should change synthetic metrics or readiness state without terminating the target Pods unless deletion is specifically the failure-boundary case.

Assert comparison invariants

Before comparing HPA outputs, assert that all non-metric variables stayed constant:

jq '{
  replicas: .spec.replicas,
  selector: .spec.selector,
  template: .spec.template.spec.containers[0].resources.requests
}' evidence/cases/complete-up/statefulset.json \
  > /tmp/complete.json

jq '{
  replicas: .spec.replicas,
  selector: .spec.selector,
  template: .spec.template.spec.containers[0].resources.requests
}' evidence/cases/one-missing-up/statefulset.json \
  > /tmp/missing.json

diff -u /tmp/complete.json /tmp/missing.json

Also compare Pod UIDs, because a same-name recreation is not the same experimental subject:

jq -r '.items[] | [.metadata.name,.metadata.uid] | @tsv' \
  evidence/cases/complete-up/pods.json | sort > /tmp/uids-a

jq -r '.items[] | [.metadata.name,.metadata.uid] | @tsv' \
  evidence/cases/one-missing-up/pods.json | sort > /tmp/uids-b

diff -u /tmp/uids-a /tmp/uids-b

Pass: only the declared metric-membership/readiness variable changes.

Fail: Pod identity, request values, HPA target, current replica count, tolerance, or stabilization configuration differs.

Status: EXPECTED / NOT EXECUTED.

Exercise failure boundaries

Acceptance needs cases that demonstrate what must not be interpreted as the documented missing-metric behavior.

The first boundary is total metric failure. If the resource-metrics API cannot provide a usable metric, HPA can expose ScalingActive=False with metric-fetch-related reasons. The Kubernetes walkthrough explains that ScalingActive indicates whether HPA can calculate desired scales and that a false value commonly indicates metric-fetch problems. That is not the same state as “some Pod metrics missing but enough information exists to calculate.”

The second boundary is multi-metric behavior. Kubernetes documents that HPA calculates a recommendation per metric and chooses the largest. If one metric cannot produce a desired replica count and the valid metrics suggest scaling down, the scale-down is skipped; scale-up can still proceed if a valid metric recommends more replicas. This playbook intentionally avoids multi-metric HPAs because that policy would add a second conservative boundary.

The third boundary is readiness timing. For CPU, Kubernetes treats certain unready Pods specially using two controller settings: the initial readiness delay, 30 seconds by default, and CPU initialization period, five minutes by default. A Pod that is merely Ready=False is not enough evidence to claim the exact branch used by HPA; start time, readiness transition history and metric timestamps matter.

Capture them:

kubectl get pods -n hpa-proof -l app=hpa-proof -o json |
jq '.items[] | {
  name: .metadata.name,
  uid: .metadata.uid,
  startTime: .status.startTime,
  ready: (
    [.status.conditions[]
      | select(.type=="Ready")
      | {status,lastTransitionTime}]
  )
}'

The fourth boundary is tolerance. HPA applies tolerance to prevent tiny metric deviations from triggering scaling; Kubernetes’ API reference documents a default cluster-wide tolerance of 10% when no per-direction value is specified. A conservative ratio of 1.066 is therefore materially different from 1.2: the former can be suppressed as “close enough” while the latter remains outside default tolerance.

Recommended boundary cases:

No production behavior should be inferred until those cases are executed under the pinned runtime.

Capture per-case evidence

A result without provenance is anecdote. Each case should be a directory that can stand alone in review.

evidence/
  baseline/
    kind-version.txt
    kubectl-version.yaml
    apiserver-version.json
    nodes.yaml
  cases/
    one-missing-up/
      input.yaml
      pods.json
      podmetrics.json
      hpa-before.json
      hpa-after.json
      statefulset-before.json
      statefulset-after.json
      events.json
      missing-metrics.txt
      oracle.json
      command-log.txt
      verdict.yaml

The ledger should make documented expectations and observations distinct:

Hash case inputs so later editing cannot silently change the conditions under which a result was obtained:

sha256sum fixture/cases/*.yaml > evidence/input-sha256.txt
find evidence -type f -print0 |
  sort -z |
  xargs -0 sha256sum > evidence/evidence-sha256.txt

Capture HPA conditions in machine-readable form:

jq '{
  currentReplicas: .status.currentReplicas,
  desiredReplicas: .status.desiredReplicas,
  currentMetrics: .status.currentMetrics,
  conditions: .status.conditions
}' "$DIR/hpa.json" > "$DIR/hpa-summary.json"

The raw artifact remains authoritative; the summary is only an index.

Preserve failure artifacts

A failed case is useful evidence only if it survives intact.

Run commands through a small harness that records the exit code without overwriting prior output:

run_capture() {
  name="$1"
  shift
  set +e
  "$@" > "${DIR}/${name}.stdout" 2> "${DIR}/${name}.stderr"
  rc=$?
  set -e
  printf '%s\n' "$rc" > "${DIR}/${name}.exitcode"
  return 0
}

Pass: stderr, stdout, exit code, raw Kubernetes objects and the case input remain available even when the assertion fails.

Fail: the harness stops at the first error and discards the failed state, or the next case overwrites evidence from the previous case.

Status: EXPECTED / NOT EXECUTED.

Reconcile state and identity

The most valuable calculation in this exercise is set reconciliation.

Define:

S = Pods selected by the HPA target selector
M = Pods represented in the Metrics API response
R = selected Pods that are Ready for this calculation
N = Pods classified not-yet-ready for CPU handling
X = S − M

The missing-metric claim is about X, not about an error message. If S={0,1,2,3} and M={0,1,2}, then X={3}. That proof should be generated from raw object identity rather than inferred from HPA’s printed average.

A reconciliation script:

python3 - <<'PY'
import json

with open("pods.json") as f:
    pods = json.load(f)["items"]
with open("podmetrics.json") as f:
    metrics = json.load(f)["items"]

selected = {p["metadata"]["name"]: p["metadata"]["uid"] for p in pods}
metric_names = {m["metadata"]["name"] for m in metrics}

missing = sorted(set(selected) - metric_names)
extra = sorted(metric_names - set(selected))

print(json.dumps({
    "selected": sorted(selected),
    "metricBearing": sorted(metric_names),
    "missing": missing,
    "extraneousMetrics": extra,
}, indent=2))
PY

For resource CPU utilization, also reconcile resource requests. A Pod with no relevant CPU request can prevent ordinary utilization calculation from meaning what the case manifest assumes; the HPA documentation explicitly ties utilization to requested resources.

This state-reconciliation technique is conceptually adjacent to Refonte Learning’s HTTP If-Match article, but the workflow is not the same. That article validates conditional HTTP writes and atomic version checks to prevent lost updates. Here, identity reconciliation is observational: no ETag or conditional update is being proposed for HPA. The relevant identity is Pod name plus UID, metric membership, selector membership and readiness state.

A replica target alone cannot tell you any of those sets.

Separate documentation from observation

There are three evidence classes in this playbook, and review should reject any assessment that collapses them.

Documented behavior comes from Kubernetes primary documentation. Kubernetes says missing metrics are set aside; on scale-down they are treated as 100% of the desired value, on scale-up as 0%; not-yet-ready Pods are also treated as 0% when the initial result points upward; direction reversal or tolerance can suppress scaling; and HPA status reports the original average utilization rather than the conservative average.

Pinned implementation evidence comes from Kubernetes v1.37.0 source. The fixture pins that version so a result can be compared with one exact controller implementation rather than “Kubernetes in general.” The v1.37.0 repository contains the replica calculator and its extensive test suite.

Observed evidence is absent until this fixture is actually executed. Therefore every case remains EXPECTED / NOT EXECUTED.

That distinction matters when documentation and observed behavior disagree. Do not rewrite the expectation after seeing output. Record the mismatch:

case: one-missing-up
documentedExpectation: 5
observedDesiredReplicas: null
status: EXPECTED / NOT EXECUTED
contradiction: null

After execution, a contradiction should look like:

case: one-missing-up
documentedExpectation: 5
observedDesiredReplicas: 4
status: OBSERVED
contradiction:
  classification: implementation-or-fixture-mismatch
  action: HOLD

Then determine whether the fixture violated an assumption: tolerance, stabilization, readiness classification, resource requests, Pod membership, stale metrics, replica count, or controller version. Only after those are excluded should an implementation-level discrepancy be escalated.

A public Kubernetes issue can also reveal implementation edge cases, but it must not replace the documented contract. For example, a 2026 issue reports cases involving succeeded Pods entering missing-Pod handling and affecting HPA calculations; that is useful as a warning that implementation details can have sharp edges, but an issue report is not a normative specification.

Define repair boundaries

A repair is acceptable only when it restores evidence collection or fixes the fixture without changing the contract being tested.

Allowed repairs include:

• fix a malformed test Metrics API response

• restore an accidentally altered CPU request to 100m

• correct a selector mismatch

• repair evidence-path or hashing scripts

• recreate the disposable cluster from the same pinned version

Changes requiring a RERUN, not a transparent repair, include:

• changing Kubernetes version

• changing HPA target utilization

• changing tolerance

• changing current replica count

• changing readiness-delay or CPU-initialization settings

• changing the expected-case arithmetic

• adding stabilization behavior

A change to the acceptance contract itself requires HOLD until review approves the new contract.

Status: EXPECTED / NOT EXECUTED.

Implement a bounded repair

The safest repair is one that is small enough to prove.

Suppose fixture-3 unexpectedly lacks a CPU request. That invalidates the intended 100m-per-Pod arithmetic. The repair is not to alter the oracle to accommodate the mistake. Repair the StatefulSet template, recreate the synthetic Pods, capture new UIDs, and rerun the entire affected case family.

Example:

kubectl patch statefulset fixture -n hpa-proof --type merge -p '
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          requests:
            cpu: 100m
            memory: 16Mi
          limits:
            cpu: 100m
            memory: 16Mi
'

kubectl rollout status statefulset/fixture -n hpa-proof

kubectl get pods -n hpa-proof -l app=hpa-proof -o json \
  > evidence/repair/pods-after-request-fix.json

Because Pod identities change after rollout, old HPA evidence is no longer comparable. Mark affected case directories superseded rather than deleting them:

case: one-missing-up
status: SUPERSEDED
reason: fixture CPU request repaired
replacementRun: run-002
releaseDecision: RERUN

The repair boundary should preserve rollback safety: the fixture must be reconstructible entirely from source-controlled manifests and the pinned node image. No production namespace, production metrics provider, or shared HPA should be touched.

A metrics-source repair follows the same rule. If the synthetic provider emits a malformed PodMetricsList, repair the provider, hash the new case payload, and rerun the complete and missing-metric comparison together. Never rerun only the failed half of a pair while retaining an old baseline unless you can prove every invariant is identical.

Verify rollback and rerun safety

Rollback for this experiment means deletion, not restoration of a production object:

kind delete cluster --name hpa-missing-metrics

That is why an isolated local cluster is preferable. Nothing in the fixture requires ownership of a production control plane.

Before deletion, archive:

tar -czf "hpa-proof-evidence-$(date -u +%Y%m%dT%H%M%SZ).tar.gz" \
  evidence fixture

A rerun should create a new run directory:

runs/
  run-001/
  run-002/

Never mutate run-001 to make it resemble run-002.

Rerun safety also means the test must not depend on a historical recommendation leaking through HPA stabilization. Kubernetes applies scaling rules after the metric recommendation; downscale stabilization defaults to 300 seconds when not overridden. That is why this fixture declares zero stabilization windows. If the test environment forbids changing that behavior, include the recommendation window in the oracle instead of pretending it does not exist.

The workflow here is deliberately separate from Refonte Learning’s CSS transition cancellation playbook. That article distinguishes transitionend from transitioncancel and itself marks unexecuted browser cases as EXPECTED / NOT EXECUTED. The shared practice is evidence preservation and explicit unexecuted status, not the lifecycle semantics themselves.

Assign operational ownership

Assign one owner to each evidence boundary:

Pass: every evidence class has a named operational owner before execution.

Fail: the same person silently edits the case payload, expected result and final verdict without review trace.

Status: EXPECTED / NOT EXECUTED.

Decide ACCEPT, REPAIR, RERUN or HOLD

The release gate should be mechanical enough that a persuasive HPA screenshot cannot override missing evidence.

The current decision for this article is:

releaseDecision: HOLD
reason:
  - fixture not executed for publication
  - no observed HPA result matrix exists
  - no per-case evidence hashes exist
  - no contradiction review has been performed
cases:
  complete-up: EXPECTED / NOT EXECUTED
  one-missing-up: EXPECTED / NOT EXECUTED
  two-missing-reversal: EXPECTED / NOT EXECUTED
  one-not-ready-up: EXPECTED / NOT EXECUTED
  missing-down: EXPECTED / NOT EXECUTED
  metrics-api-503: EXPECTED / NOT EXECUTED

The decision flow is:

The central acceptance rule is simple: never infer missing metrics from the replica target alone. Prove missingness by reconciling the selected Pod set against the raw PodMetrics set, then compare the controller’s decision with an independent conservative recomputation.

That is also why kubectl get hpa is evidence but not an oracle. Kubernetes explicitly says that the status average can remain the original average even when conservative recomputation affects the decision.

Connect the practice to Refonte Learning

This workflow sits naturally inside DevOps engineering because it combines Kubernetes controller behavior, metrics plumbing, reproducible environments, evidence capture, failure analysis and release gating rather than treating autoscaling as a single YAML feature.

Refonte Learning’s DevOps Engineering Program page currently describes a three-month program with an indicated dedication of 12–14 hours per week, covering Linux, Git and GitHub, CI/CD, Docker and Kubernetes, Terraform, cloud platforms, and monitoring/logging; it also lists potential internship opportunities and career-result categories such as DevOps Engineer, Cloud Engineer and Site Reliability Engineer. The broader programs catalog lists DevOps Engineer in its Cloud & Security category and identifies it as an intermediate, three-month program.

For engineers who want structured practice around Kubernetes, CI/CD, infrastructure automation and monitoring, the program page is the appropriate place to evaluate whether its current DevOps offering matches their goals.

Until the fixture is actually run and its ledger populated, the correct release decision remains HOLD: EXPECTED / NOT EXECUTED.