Beyond the Warm-Cache Demo: Test HyperPod Autoscaling Under Real Conditions

The operating decision is not whether a warm-cache demo looks faster. It is whether HyperPod model caching supports the service when placement is favorable, when a replacement node is empty, and when scale-out exceeds the set of already-warmed nodes. A cache that improves the best case can still leave the service exposed to cold source reads, image pulls, storage pressure, and recovery delays.

AWS announced general availability on September 11, 2026, reporting around 60% faster scale-out for models from 57–145 GB and a 97% reduction in container-image pull time in its benchmark scope. Those are vendor-reported benchmark results, not universal service-level guarantees. AWS’s GA announcement, dated September 11, 2026, describes two independent mechanisms: model-weight caching and container-image pre-pulling.

This playbook is for AI platform teams, ML engineers, infrastructure owners, and FinOps partners evaluating HyperPod inference autoscaling. The outcome is an evidence package that separates image and weight behavior, distinguishes warm from cold placement, proves replacement-node recovery, measures local capacity pressure, and defines workload-specific service and cost gates. The acceptance standard is deliberately stricter than “the cache hit was fast”: the deployment must remain correct and operational when the cache is absent.

Define the scale-out objective the cache must support

Start with a service contract, not a cache switch. Define the approved model identity, the traffic shape that causes scale-out, and the point at which added capacity becomes useful. “Pod created” is not a serving objective. A more defensible boundary is the time from the chosen scale-out trigger to the first successful request through the newly added replica, followed by evidence that the replica serves the approved model correctly under the expected request mix.

Assign owners before the run. The ML owner approves model identity and response correctness. The platform owner owns placement, cache state, and readiness evidence. The service owner selects the time-to-serve service-level objective (SLO) and acceptable error behavior during expansion. FinOps owns the billing inputs and cost comparison. This is narrower than a general production machine learning deployment discussion: here, the acceptance contract is specifically about scale-out under different cache states.

Proposed acceptance contract

The empty-node row is non-negotiable for this evaluation model. AWS documents that a replaced node must be warmed again before it becomes a preferred warm target; local cache state is not presented as durable state that follows the node. AWS’s live model-caching documentation, accessed September 15, 2026, explicitly describes re-warming after node replacement.

A deployment that meets the SLO only when the scheduler finds an already-warm node has demonstrated a useful optimization, not a production-ready failure envelope. Your acceptance contract should therefore contain both a warm-path target and a cold replacement-node target, even if the two thresholds differ.

Separate the cache types and the benchmark claims

“Model cache” is too broad for incident analysis. HyperPod exposes separate controls for model weights and container images, and AWS documents both as disabled by default. Weight caching pre-populates model files on host-local non-volatile memory express (NVMe) storage; image caching pre-pulls the inference server image to target nodes. They can be enabled independently or together through modelCacheConfig. AWS’s live caching documentation, accessed September 15, 2026, documents the two controls and their defaults.

AWS also documents a separate HyperPod key-value (KV) caching and intelligent-routing feature for reusable inference state. That feature concerns precomputed key-value vectors and is conceptually different from host-local model weights and container images. AWS’s live KV caching documentation, accessed September 15, 2026, should be treated as a separate serving optimization.

Model-weight and image caches solve different loading steps

The HyperPod weights cache addresses one loading interval: obtaining model files from the configured model source. The HyperPod container image cache addresses another: obtaining the inference server image from its registry. AWS states that a cache miss falls back to the original source, and its live documentation describes preferred placement on weight-cache-ready nodes rather than a hard requirement that every pod land on one.

Keep four timestamps separate: image availability, weight availability, application/model readiness, and first successful request. Even if both caches hit, the inference process can still require framework startup, model initialization, memory allocation, sharding or worker coordination before it is actually ready. Those intervals are workload-specific; do not attribute them to caching unless your instrumentation shows the causal boundary.

Vendor benchmark percentages are hypotheses for your workload

AWS’s September 11 announcement reports “around 60% faster scale-out” across 57–145 GB models and a 97% image-pull-time reduction in its tests. The September 10 AWS Machine Learning Blog similarly reports around 60% faster scale-out with weights caching and up to 97% image-pull reduction.

For model caching benchmark methodology, reproduce the comparison using your model size, image, model source, node type, network path, replica count, and traffic trigger. Do not translate a component-level percentage into the same percentage for request latency, throughput, availability, or spend. A 97% image-pull reduction can be operationally important while producing a much smaller end-to-end improvement if weight loading or model initialization dominates the path.

Inventory instance storage and deployment requirements

The first hard gate is storage support. AWS’s live documentation says model-weight caching requires local NVMe at the configured host path, with /opt/dlami/nvme as the documented default, and that EBS-only instances are not supported for the weights cache. It also says the maximum cacheable model size is bounded by available local NVMe capacity.

That is a product-specific prerequisite, not a generic statement that “fast cloud block storage is good enough.” Record what the target node actually exposes. Verify the node group or instance type, mount point, free space, model size, and which other deployments may consume the same local device. The same discipline used in broader cloud development practices is useful context, but the NVMe requirement here comes from AWS’s HyperPod documentation, not from a general architecture pattern.

Pre-test inventory

Confirm the HyperPod Inference Operator version as part of the inventory. AWS’s HyperPod Inference release notes list operator v3.3, released August 4, 2026, as introducing host-local model caching; the public “What’s New” GA announcement is dated September 11, 2026. These are different dates for different publication/release milestones, not evidence that one source invalidates the other.

Also record the no-cache baseline. Without it, an apparent improvement may reflect a changed image layer, a faster source path, a different node type, or background network conditions rather than the cache configuration itself. Hold those variables constant where practicable and record exceptions when they cannot be held constant.

Build a cache-state and placement test matrix

The core evaluation is a proposed matrix, not an AWS-prescribed certification suite. Cross cache configuration with placement state and failure condition so that favorable warm runs cannot hide the cold path. Leave result cells blank until your team executes the tests.

Run the matrix in a non-production or safely isolated environment first. A failure-injection test should never jeopardize unrelated workloads or production artifacts. Define the blast radius, stop condition, and restoration step before making a source unavailable or deliberately tightening storage headroom.

There is also a documentation-scope tension worth testing rather than smoothing over. The September 10 AWS blog says the weights-cache flow waits for all target nodes to become cache-ready before creating the initial inference deployment, while the live product documentation describes runtime preferred placement and source fallback when a pod reaches a node without the cache. Treat those statements as different lifecycle contexts and verify the behavior of your installed operator version.

Warm placement is a preference, not an acceptance guarantee

AWS documents that weight-cache-ready nodes receive labels and the deployment uses preferred node affinity so pods prefer warm nodes but can schedule elsewhere. On a node where the cache is not available, the deployment falls back to the remote model source.

That maps cleanly to general Kubernetes scheduler language but should not be replaced by it. Kubernetes “Assigning Pods to Nodes,” live documentation accessed September 15, 2026, distinguishes required constraints, which filter eligible nodes, from preferred constraints, which influence scoring.

For Kubernetes cache placement, record both layers: the deployment’s hard eligibility constraints and the cache-related soft preference. If a test unexpectedly lands cold, first ask whether no warm eligible node existed, whether another hard constraint excluded it, or whether concurrent demand consumed the warm capacity. Do not label all such outcomes “scheduler failure.”

A replacement node must pass with an empty local cache

AWS says that after a node replacement, the operator must warm the cache on the new node again before pods are preferentially scheduled there. Existing warm nodes can continue serving in the meantime.

Your replacement-node inference testing should therefore assume zero useful local cache on the new machine. Start the clock at the agreed replacement or scale-out event. Record image acquisition, weight acquisition, model readiness, and the first successful request separately. Verify the model identity after startup.

The pass condition should not be “eventually recovered.” It should be an owner-selected cold-path SLO plus correctness. If an empty node cannot reach service within the accepted recovery window, warm-node performance does not cancel that risk. Either change capacity/placement architecture, revise the service objective explicitly, or hold rollout.

Budget local capacity before optimizing speed

Local NVMe is now part of the serving capacity model. AWS documents that each deployment uses an isolated per-deployment cache directory and that multiple cached deployments on the same node each consume storage. Its troubleshooting guidance explicitly calls out insufficient disk and multiple deployments causing disk pressure.

Build an NVMe model cache capacity worksheet before enabling caching broadly. Use observed sizes rather than parameter-count estimates because packaging, quantization, tokenizer assets, auxiliary files, and deployment duplication affect actual bytes on disk.

Proposed capacity worksheet

Do not invent a universal “safe” free-space percentage. Kubernetes exposes disk-pressure signals such as nodefs.available and imagefs.available, and kubelet behavior depends on configured eviction thresholds and filesystem layout. The live Kubernetes node-pressure eviction documentation, accessed September 15, 2026, describes DiskPressure, reclamation, image garbage collection, and possible pod eviction after reclamation is insufficient.

Use your cluster’s actual kubelet configuration and node filesystem layout as the policy source. Then choose an application safety margin above the point at which platform stability becomes uncomfortable. That margin is a local operating policy, not an AWS or Kubernetes universal.

Capacity also changes during updates. If old and new artifacts coexist during a transition, the peak local requirement can exceed steady state. Measure the actual overlap behavior for your deployment path rather than assuming a single-copy budget.

Keep artifact identity and provenance visible

A faster start is a failed optimization if it serves the wrong weights. Treat cache correctness as an artifact-provenance problem first and a performance problem second. Every run should bind the deployment revision, model source, immutable artifact identifier, image identity, and verification result into one record.

The AWS Machine Learning Blog states that source updates at the same Amazon S3 path are not automatically detected if the InferenceEndpointConfig specification does not change; it recommends updating the specification, for example by changing the path or adding a version suffix. That makes mutable “latest” paths particularly poor evidence for a cache test.

Proposed provenance record

Keep responsibility explicit. A platform engineer may prove that bytes came from the local cache while an ML engineer proves they are the approved bytes. That distinction also fits the broader separation of AI and machine learning engineering roles, but the acceptance evidence here should name your real operational owner rather than infer ownership from a job title.

For model updates, prefer an immutable source/version that makes a changed artifact visible in the deployment specification. Validate the new model on both a warm path and an empty replacement node. Then verify that the previous deployment remains recoverable until the new revision passes its rollout gates.

A source-read failure needs the same provenance discipline. An old cached copy may still be locally readable, but “there were bytes on disk” is not proof that the cache contains the currently approved model or that a new node can recover. Your incident decision should use model identity, approval state, and source recoverability together.

Instrument the serving timeline consistently

Do not collapse cold start into one timestamp. For operational diagnosis, separate scheduler delay, image acquisition, weight acquisition, application initialization, readiness, and the first successful request. The cache can materially improve two of those intervals without eliminating the others.

Use a monotonic clock for durations when your instrumentation supports it, and use synchronized wall-clock timestamps when correlating events across nodes and control-plane components. The exact metric source is implementation-specific; AWS’s documentation gives cache-ready labels, pod inspection, and operator logs as verification mechanisms, but it does not define a universal end-to-end metric schema for your application.

Blank serving-timeline template

General observability fundamentals can help structure traces, metrics, and logs, but the fields above should be tied directly to the deployment’s acceptance contract rather than collected because a telemetry platform happens to expose them.

Do not infer a cache hit solely from a fast duration. Correlate time with cache-ready state, node identity, mount/source evidence, and relevant events. Conversely, a slow warm-node run may expose model initialization, GPU allocation, contention, or application readiness problems that caching does not solve.

For repeated runs, preserve per-replica data rather than only averages. Tail behavior matters during burst scale-out: one cold pod among several warm pods can become the limiting added replica if traffic distribution or readiness logic depends on all replicas becoming available.

Work through warm, cold and replacement-node scenarios

Use one deployment revision and one traffic replay to compare states before changing other variables. The worked example below is a proposed exercise. It intentionally contains blank measurements; no successful outcome is implied.

This table is the practical center of warm vs cold scale-out. It stops the team from presenting one “cache-on” average that mixes fundamentally different conditions. Repeat enough runs to understand variance under your traffic and infrastructure conditions; the required sample count is an owner decision based on risk, not a universal number supplied by AWS.

Compare favorable and unfavorable placement

Group results by observed state, not by intended state. A run configured with caching enabled is not necessarily a warm-cache run. AWS documents that pods can schedule where the cache is not available and then use the original source path.

Report warm-existing-node, empty-node, and mixed/constrained-placement distributions separately. If both caches are enabled but only the image is ready on the selected node, classify the run accordingly. Preserve outliers and explain them before aggregating.

The useful comparison is causal: which interval changed when one cache state changed and the rest of the deployment stayed comparable? That is stronger evidence than “total startup looked better on Tuesday.”

Evaluate model updates and failed source reads

Add an immutable updated artifact and rerun the relevant rows. Confirm that the served model identity changes to the approved new version; do not treat reuse of an old local cache as a successful update. AWS’s September 10 blog notes that changing the model source causes a new cache/rollout, while a same-path content change without a spec change is not auto-detected.

Then perform a controlled source-read failure in an isolated test environment. Test both a warm node and an empty node. The warm node may reveal what remains locally usable; the empty node proves whether recovery still depends on the source. Record the failure mode rather than assuming the cache is a source-availability substitute.

The recovery requirement is explicit: restore a valid source-load path and maintain a tested prior deployment. Cached bytes on a surviving node can be useful during an incident, but they are not a durable recovery contract for node loss.

Test concurrent scale-out and capacity pressure

Single-replica tests understate the resource competition created by real bursts. Model cache cold-start testing should include concurrent scale-out, competing cached deployments, and at least one planned node replacement while local storage is within an owner-approved test margin.

AWS notes that multiple cached deployments consume local storage independently and recommends separate instance groups or reducing concurrent cached deployments when they cause disk pressure. Kubernetes, separately, documents that DiskPressure can lead the kubelet to reclaim resources, including unused images in applicable filesystem layouts, and potentially evict pods if reclamation is insufficient. These are related operational concerns, but do not assume HyperPod weight-cache files and container-runtime image garbage collection have identical lifecycle semantics.

Use your existing telemetry stack to retain the evidence. The monitoring and logging foundations article is useful general context for collecting metrics and logs; the concrete signals for this experiment should still come from the test plan and your cluster configuration.

Controlled concurrency checklist

• Request the owner-approved number of additional replicas: ______; record each pod’s node and cache state.

• In parallel, warm or update a second deployment with known local-storage demand: ______.

• Introduce one planned replacement-node event and verify the new node begins without the old local cache.

• Record free capacity, DiskPressure, image/weight warm-up errors, evictions, and per-replica first-success times.

• Stop if unrelated workloads are threatened, model identity becomes uncertain, or the configured safety floor is crossed.

Do not call the run successful merely because all pods eventually become ready. Look for synchronized source downloads, contention that stretches tail startup time, a warm-node shortage that forces fallback, or pressure behavior that makes a future scale-out less predictable.

For a constrained system, an acceptable outcome may be fewer cached deployments per node, dedicated instance groups, a larger-NVMe instance type, or a deliberately smaller warm set. Those are architecture choices to validate against service and cost goals, not universal prescriptions.

Compare service outcomes and cost honestly

Caching earns a rollout only if the service outcome justifies the operational and resource tradeoff. Keep the warm and cold populations separate, then compare them against owner-selected SLOs. The service owner should care about successful capacity becoming available, not only image-pull duration; FinOps should care about measured billable resource time and verified rates, not an assumed percentage copied from a benchmark.

Proposed service-and-cost gate

A useful GPU serving cache cost model is simple in form but must use current inputs: measured billable resource duration multiplied by the applicable verified rate, plus any relevant source, storage, or transfer charges your architecture incurs. Do not hard-code a price into the playbook. Rates, commitments, allocation rules, and chargeback practices can change.

A 97% reduction in isolated image-pull time does not imply a 97% reduction in cost or end-to-end scale-out time. If weights, model initialization, scheduling, or traffic ramp dominates, the business impact can be much smaller even though the image optimization works exactly as designed. AWS’s percentages should therefore remain component evidence and hypotheses for your environment, not cost multipliers.

FinOps should also see the cost of keeping the architecture cache-capable: instance-type choices, any capacity held for recovery, and operational reruns. Attribute only costs that your design actually creates; do not assume caching itself requires a fixed amount of extra GPU idle time.

Roll out with a tested cold-path fallback

A rollout should narrow exposure while preserving a path that does not depend on surviving local cache state. Start with a constrained serving cohort, confirm both warm and empty-node behavior, and expand only after the same revision passes correctness, service, capacity, and recovery gates.

AWS documents remote-source fallback when a pod lands where the weight cache is unavailable. That behavior is useful, but “the product can fall back” is not the same as “our source path is reachable, authorized, fast enough, and tested under incident conditions.” Validate the source-load path deliberately.

Proposed rollout and recovery gates

A complete rollback is more than disabling a cache flag. Maintain a prior deployment revision whose model source, image, configuration, and serving path have been exercised recently enough to be credible. Preserve access to the source of truth. Do not promise that local files survive hardware replacement.

Treat this as part of the operational lifecycle rather than a one-time launch step. The broader DevOps lifecycle management model is useful for change control and feedback loops, while this HyperPod-specific gate adds the cache-state evidence required for inference.

When recovering, prefer an explicit sequence: stop expansion, protect currently healthy capacity, verify the approved artifact and source path, restore or roll back the deployment, then repeat the failed acceptance row. A warm-cache success after the incident is not sufficient if the original failure occurred on an empty node.

Maintain cache-readiness evidence over time

Cache readiness drifts because deployments, artifacts, node groups, source paths, and cluster policies change. Treat the original benchmark as evidence for one revision and one environment, not a permanent property of the service.

AWS’s live documentation provides concrete operational checks: cache-ready node labels, pod inspection for the host-local cache mount, and inference-operator logs for warm-up activity or errors. Those checks can support a runbook, but your service still needs higher-level evidence for correctness and time to first successful request.

Proposed drift-review register

Choose a review cadence based on change rate and service criticality. The important rule is event-driven reruns after material changes, plus a periodic check frequent enough that the prior-deployment recovery path has not silently decayed.

A second operator should be able to reproduce the evidence without relying on tribal knowledge. Store the deployment revision, immutable model identifier, node-group definition, cache settings, traffic replay description, owner thresholds, raw timestamps, and verdict together. Keep secrets and production identifiers out of shared templates.

Watch artifact growth as a first-class capacity trend. A deployment that fit comfortably last quarter may cross the local headroom boundary after a larger model, different quantization package, or additional cached deployment. The inference rollout acceptance gates should reopen when that assumption changes.

Finally, retain failed runs. They are often more valuable than the clean benchmark because they document the actual fallback path, source dependency, and pressure behavior that future operators need during an incident.

Pilot the cache without hiding the cold path

A 30-day pilot is a proposed operating schedule, not AWS guidance. Its purpose is to prevent a quick warm-cache win from becoming the only evidence presented at rollout review.

Before the limited cohort expands, require the replacement-node gate to pass. Also require a tested source-load path and a prior deployment that can be restored. A cohort should not grow merely because warm-node p95 looks good.

Keep the final pilot review compact. Ask whether the approved model was served in every accepted state, whether cold placement met its explicit objective, whether NVMe headroom remained inside the owner-selected safety boundary, whether concurrent scale-out behaved predictably, and whether measured cost supports the operating choice. Any unresolved “no” should remain a rollout hold rather than becoming an undocumented exception.

For practitioners building the underlying skills to design this evidence, the Refonte Learning AI Engineering Program lists model development and optimization, data engineering for AI, scaling AI systems, ethics and governance. It runs for three months at 12–14 hours per week, with a stated requirement of pursuing or completing a bachelor’s degree in computer science, engineering, mathematics, or a related field. This is foundational training, not a claim that HyperPod or model-caching labs are in the curriculum.

Answer the caching questions and issue a serving verdict

The final review should be readable without replaying the whole benchmark. Use four questions to remove the most common category errors, then issue one serving verdict.

Is HyperPod model caching the same as inference KV caching? No. Model-weight caching stores deployment weights on host-local NVMe, and image caching pre-pulls the inference server image. HyperPod’s separate KV caching feature concerns reusable inference key-value state. Treat them as different mechanisms with different measurements.

Does warm placement guarantee a cached node? No. AWS documents preferred placement on cache-ready nodes and fallback when the cache is unavailable. General Kubernetes documentation likewise distinguishes preferred scoring from required filtering.

Does a replacement node inherit the local cache? No documented guarantee says that it does. AWS explicitly says the operator must warm the cache on a replacement node again before pods are preferentially scheduled there. Design recovery as if the replacement starts cold.

Should we expect AWS’s 60% and 97% figures? Treat them as vendor benchmarks, not promises. AWS reports around 60% faster scale-out for tested 57–145 GB models and a 97% image-pull-time reduction in the announced scope. Your result depends on model size, image, source, node, network, placement, concurrency, and the interval you measure.

Serving verdict gate

Mark the deployment ready only when correctness, cold-path service, local capacity, concurrency, recovery, and cost gates pass under the workload-specific thresholds your owners approved. Mark it hold for incorrect model identity, unexplained cold-path failure, inadequate local capacity, or an SLO that passes only with warm placement. That verdict is the production value of HyperPod model caching evaluation: not proving that a warm cache can be fast, but proving the service remains trustworthy when the cache is not there.