GKE Can Suspend an Agent’s RAM and Resume It in Under a Second: Evaluate Agent Substrate Safely

Google Cloud introduced Agent Substrate on GKE through its September 11, 2026 release note. The capability suspends an idle agent, snapshots active RAM and local files, and restores that state onto an available sandbox in under a second. That is an important density mechanism for interactive agents that spend most of their lives waiting, but it is not proof that a resumed agent is the same logical session, holds the right identity, has reconciled an outstanding tool call, or can safely repeat an external action.

The availability boundary is equally important. The current dedicated Agent Substrate documentation makes the feature available for evaluation and non-production use, while production support is restricted to an allowlisted limited-GA program. That status should prevent a benchmark win from becoming an unreviewed production standard. Broader Kubernetes platform trends explain why specialized runtimes are emerging, but they do not answer whether this runtime preserves an individual agent correctly under suspension, node loss, version change, or a simultaneous wake-up of thousands of sessions.

This guide treats GKE Agent Substrate as a state machine and a security boundary, not as a feature announcement. It defines a representative agent, the checkpoint boundary, an isolation threat model, comparative benchmarks, failure injections, measurable gates, and a reversible pilot. The required outcome is evidence that idle-density savings survive the full cost of warm capacity, snapshot storage, reconnection, observability, support constraints, and recovery. Sub-second restore matters only after session correctness, tenant isolation, side-effect safety, and an explicit support path have been proven.

1. Define the Workload and the Decision Agent Substrate Must Improve

Start with a decision statement that can fail. A useful form is: “For qualified interactive agents with an idle ratio above 80 percent, suspension will reduce the cost per successfully completed active session by at least 25 percent, while meeting the same correctness, isolation, and recovery gates as the current platform.” The percentages are practitioner-selected hypotheses, not vendor guarantees. Set them from your own baseline before anyone sees the substrate result.

The unit of evaluation must be a workload, not an actor count. A million registered actors can look impressive while only a small fraction resume correctly, downstream systems collapse under reconnects, or the operations team carries an unacceptable recovery burden. Define the business task, arrival pattern, session duration, idle interval, state size, tool dependencies, and consequence of duplication. If the agent performs irreversible financial, legal, safety, or access-control actions without a reliable idempotency and reconciliation design, it is not an early pilot candidate.

Use a reference agent that is deliberately stateful. In this guide, the reference agent coordinates a synthetic fulfillment case. It keeps an in-memory plan and conversation cursor, writes a draft artifact and tool receipt to local files, stores the authoritative case status in an external database, calls a mock ticketing service, schedules a follow-up timer, and emits an external “case created” side effect. Every request carries tenant, user, actor, workflow, deployment revision, and state-epoch identifiers.

Establish two controls before installing anything: a conventional Pod deployment that externalizes state, and the closest suitable Agent Sandbox configuration. The comparison should use identical application code, data, identity scopes, network path, and tool mocks. Existing Kubernetes and platform-engineering foundations remain prerequisites because a specialized runtime does not remove responsibility for cluster versioning, node pools, storage, IAM, network policy, image provenance, or incident response.

Qualify the workload only when all of these statements are true:

• The agent is interactive or event-driven, with enough idle time for suspension to matter.

• Local process state is valuable to preserve, but durable business truth already has an external authority.

• Tool calls support idempotency, status lookup, or compensating action.

• The application can detect a stale, incompatible, or wrong-tenant restore before accepting work.

• The team can route new sessions back to the control platform without waiting for snapshot conversion.

Disqualify the workload when it requires preserved open sockets, depends on live GPU context, cannot distinguish completed from uncertain side effects, or has regulatory rules that prohibit the proposed snapshot retention model. A density feature should not be used to hide a missing application state contract.

2. Confirm Availability, Support, and Evaluation Boundaries

Availability is part of the architecture. Record what the vendor supports, what the open-source project provides, and what your own team is prepared to operate. Do not use the word “GA” without its qualifier: limited-GA production support on an allowlist is not the same operating promise as general availability for every customer, region, configuration, and workload.

2.1 Separate Evaluation From Allowlisted Production Support

Official behavior: As verified on September 14, 2026, Agent Substrate is available to Google Cloud customers for evaluation and non-production use. Production support is offered only on an allowlist through a limited-GA program. The generic Google Cloud launch-stage definitions explain that preview and legacy pre-GA stages normally lack broad SLA and support commitments unless specific terms say otherwise. For this product, the dedicated page and the written terms of your allowlisted program should govern the decision.

Practitioner requirement: Create a support record before any production authorization. It should name the accepted project and cluster, covered versions and locations, severity path, support hours, escalation contacts, known exclusions, upgrade expectations, and the evidence Google requests during an incident. A signup submission or successful installation is not support confirmation. Store the acceptance artifact with the architecture decision record.

First-party pages can age differently. The release-note entry is the historical announcement, while the dedicated product page was updated September 10 and the installation page September 11. The dedicated pages include limitations that an aggregate release feed cannot safely replace. If status text conflicts, freeze expansion, ask the program contact for a written interpretation, and use the narrower support position until the conflict is resolved.

2.2 Record Region, Cluster, Version, and Quota Constraints

Build an environment ledger from the current GKE documentation and the Agent Substrate installation requirements. Record the project, region or zone, GKE mode and version, node image, machine series, CPU architecture, Workload Identity pool, snapshot bucket, network plane, quota ceilings, substrate version, ActorTemplate revision, and rollback target. Capture this ledger for every benchmark run so that a later result is reproducible.

The current constraints are material. Agent Substrate is for GKE Standard, not Autopilot. It requires GKE 1.36 with the specified beta certificate APIs enabled at cluster creation, or GKE 1.37 or later. Versions before 1.36 are unsupported. Existing 1.36 clusters cannot be retrofitted if the required beta APIs were not enabled at creation. Workload Identity Federation and Cloud Storage are required. Mixed-architecture machine series such as E2 are unsupported, one ActorTemplate must target a uniform VM type, and live GPU device use is not supported for Actor containers.

The current Agent Substrate installation guide adds further operating facts: an existing cluster must meet the documented machine-size requirement; the installer creates or configures a snapshot bucket, IAM bindings, monitoring, custom resources, certificate components, and a PostgreSQL database that tracks Actors and Workers. Components run as ordinary workloads in dedicated namespaces and do not modify the GKE control plane. Agent Substrate itself carries no additional GKE product charge, but every underlying resource is billable and operationally yours.

Acceptance record for this section:

• Written availability and support status, including any allowlist approval identifier.

• Exact cluster, node, substrate, image, and application versions.

• Region and quota inventory, including token exchange, Cloud Storage, compute, network, and downstream-service limits.

• Known product limitations mapped to application behavior and test cases.

• Named owners for cluster, runtime, application state, security, capacity, and vendor escalation.

3. Model Agent State Across Memory, Files, and External Systems

A runtime snapshot is not an application memory architecture. The snapshot can preserve process RAM and local files, while durable user facts, authoritative workflow status, vector memory, external tool state, and audit records remain separate systems. The distinction aligns with established AI agent memory architectures: working state can accelerate continuity, but it must not silently become the only source of truth for long-lived business facts.

Define a checkpoint envelope at the application layer. It should include the tenant ID, actor ID, session ID, workflow ID, deployment revision, state schema version, monotonically increasing state epoch, last committed tool operation, outstanding idempotency key, timer deadlines, and checksums for deterministic state and critical local files. The envelope can live in local state for fast inspection, but copy its commit marker to durable storage so the resumed actor can detect rollback or divergence.

The restore sequence should be defensive: authenticate the incoming request, read the authoritative actor-to-tenant mapping, compare the restored envelope with durable state, reject incompatible versions, refresh credentials, reconnect dependencies, reconcile uncertain tool calls, recalculate timers, and only then mark the actor ready. A process that responds to traffic before this sequence finishes has optimized the wrong latency.

Use explicit states such as ACTIVE, SUSPENDED, RESTORING, RECONCILING, READY, QUARANTINED, and RETIRED. The application does not need to duplicate the substrate controller, but it needs a logical model that explains which requests are allowed and what evidence exists at each boundary.

4. Threat-Model the Sandbox, Snapshot, Identity, and Network

Treat the system as several trust boundaries: the caller and routing layer, the actor identity registry, the worker sandbox, the snapshot store, the cluster and node, the credential path, the egress path, and each external tool. The official GKE security overview emphasizes layered controls across image, runtime, network, identity, and control-plane access. Adjacent cloud workload security practices provide useful context, but this pilot needs tests that are specific to suspend and restore.

4.1 Test Cross-Tenant and Cross-Session Isolation

Sandbox isolation and platform multi-tenancy are different claims. A kernel isolation runtime can constrain code relative to the host while the routing, identity, snapshot naming, storage permissions, telemetry labels, or application cache still mixes tenants. Test every transition where a tenant identifier is accepted, derived, stored, or restored. The expected tenant must come from authenticated authorization context, not from a mutable field inside the snapshot.

Create at least two synthetic tenants with identically shaped actors, filenames, tool names, and session numbers. Attempt to send tenant A requests to tenant B actor IDs, copy or rename snapshot references, replay stale routing tokens, reuse worker slots, collide cache keys, and restore after deleting one tenant. Instrument canary values that must never appear outside their tenant. Run enough randomized attempts to exercise concurrency, not only a scripted happy path.

The acceptance gate is zero cross-tenant disclosure or action. Do not convert this to a percentage SLO. One confirmed event stops the pilot, quarantines affected snapshots and workers, preserves evidence, rotates exposed credentials, and triggers an access review. Resume only after the root cause, affected scope, cleanup proof, and regression test are documented.

4.2 Protect Secrets, Tokens, and Snapshot Artifacts

Assume RAM and local files can contain prompts, retrieved documents, API responses, environment variables, bearer tokens, private keys, and generated code. Minimize what enters the snapshot. Prefer workload identity and short-lived credentials obtained after resume, store durable secrets outside the worker, avoid plaintext token caches, and scrub temporary files before the agent becomes idle. Encryption at rest is necessary, but it does not correct excessive IAM or indefinite retention.

Agent Substrate requires Workload Identity Federation for GKE. Its credential flow exchanges a Kubernetes service-account token for a short-lived federated token. Current documentation also warns about identity sameness: matching namespace and service-account identities in clusters that share a workload identity pool can be treated as the same principal unless projects, names, or IAM conditions separate them. Token exchange currently has a documented request-per-minute quota, and cached access tokens can approach expiry. Test both mass refresh and resume with an expired token; custom clients must refresh rather than replay failure indefinitely.

Network assumptions need special attention. Agent Sandbox documents a default-deny posture with explicit egress rules, while the current Agent Substrate limitations state that its EgressPolicy capabilities, including default deny, hostname rules, and credential injection, are not supported. Do not infer equivalent controls from the product names. Enforce the approved path through cluster network policy, VPC and firewall design, private service access, an authenticated egress proxy, destination controls, and tool-side authorization, then verify the result from inside a restored worker.

Define snapshot retention by data class and legal need. The deletion workflow must cover the primary bucket, versions, retention locks, replication, backups, derived crash artifacts, and telemetry. Prove that retiring an actor prevents future restore and that a tenant deletion request has an auditable completion state. If the platform cannot meet the required deletion time, the workload is not eligible even when runtime isolation passes.

5. Build a Representative Stateful Agent Benchmark

A counter demo proves that an integer can survive suspension. It does not represent an agent that has local files, durable state, an in-flight tool call, a timer, and an external side effect. Build a benchmark agent with all five. Keep the external systems synthetic so failure injection is safe, but preserve production-like latency, authentication, retries, payload sizes, and idempotency semantics.

The reference fulfillment agent receives a case, loads tenant policy, creates a local plan, writes a generated report, updates an authoritative case row, calls a mock ticket service with an idempotency key, and schedules a follow-up. The tool service records request receipt and completion separately, which lets the benchmark create the dangerous interval where the remote side effect succeeds but the local acknowledgment is not yet committed. A checksum covers the plan, cursor, local file manifest, tool ledger, timer deadline, and state epoch.

Store one machine-readable evidence record per run. The following is an internal benchmark record, not an Agent Substrate API or configuration format.

run_id: bench-004281
scenario: resumed
expected_tenant: tenant-a
expected_actor: actor-017
application_revision: app-r42
state_schema: 3
pre_suspend_epoch: 118
post_resume_epoch: 118
state_checksum_match: true
local_file_manifest_match: true
uncertain_tool_operations: 0
duplicate_external_side_effects: 0
network_reconnects: 3
resume_latency_ms: 742
end_to_end_task_latency_ms: 1386
result: pass

Use a fixed corpus and seed for comparative runs, then add stochastic variation for robustness. Preserve the same actor count, state-size distribution, request schedule, tool behavior, and node budget across standard Pods, Agent Sandbox, and Agent Substrate. Report each scenario independently; a combined average can hide the precise mode that makes production unsafe.

6. Measure Suspension, Restoration, and End-to-End Correctness

Measure a timeline, not one latency. At minimum, record trigger receipt, route decision, worker claim, snapshot read start and finish, process restoration, application reconciliation start and finish, readiness, first response byte, tool completion, and final business result. “Resume latency” should mean the platform interval you explicitly define. “End-to-end latency” should include everything the user waits for, including credential refresh, socket reconstruction, tool reconciliation, and any queue.

For every scenario, publish p50, p95, p99, maximum, confidence interval, sample count, state size, and failure rate. Separate warm-cache from cold-cache runs, short from long idle periods, and small from large snapshots. A sub-second median can coexist with a damaging p99, and one successful restoration cannot support a reliability claim.

Correctness needs multiple checks. Compare the application state checksum and critical local-file manifest before suspension and after restoration. Confirm the expected tenant and actor mapping from durable authority. Verify the state epoch did not move backward. Reconcile each tool operation against the external ledger. Execute a deterministic next step and compare its result with the unsuspended control. Finally, verify the user-visible task outcome rather than stopping at a healthy process or successful route.

A measurable starting gate for a non-production pilot is: 100 percent checksum and file-manifest parity for deterministic fields; zero wrong-tenant restores; zero duplicate irreversible side effects; no silent acceptance of incompatible state; failed resumes below 0.1 percent across at least 10,000 representative resumes, with every failure entering a tested recovery path; and p95 restore below one second where the vendor claim applies. End-to-end task latency must separately meet the product SLO. Adjust sample size and thresholds to risk, but never relax the zero-tolerance security and corruption gates.

When a run fails, preserve the pre-suspend envelope, snapshot identifier, worker and node metadata, application and substrate versions, timing trace, tool ledger, and relevant logs. Do not automatically retry until evidence is captured, because the retry can overwrite the state needed to understand the first failure. A retry that eventually succeeds is still a failed resume for reliability accounting.

7. Test Timers, Tool Calls, Sockets, and Side Effects

Suspension freezes local execution, but wall-clock time and external systems continue. Define timer semantics by deadline, not by remaining process duration. On resume, compare the stored deadline with current time and choose an explicit policy: execute once, skip, reschedule, or require human review. Test daylight-saving changes where relevant, clock skew, long suspension, and a burst of overdue timers. Never allow every missed timer to fire immediately without a capacity and consequence policy.

Open connections are not preserved. The application must recreate database pools, DNS resolution, TLS sessions, MCP or tool connections, and streaming channels. Test endpoint changes during idle time, certificate rotation, revoked credentials, connection limits, and a dependency that is unavailable at resume. Readiness should remain false until required connections are either healthy or the agent has entered a defined degraded mode.

Worked failure-injection example: The agent writes an idempotency key and “tool call pending” record, then asks the mock ticket service to create a case. The service commits the case and returns success, but the benchmark drops the response and triggers suspension before the agent records completion. After resume, a naive agent repeats the call. The correct agent queries the service by idempotency key, finds the completed case, records the remote identifier, advances the state epoch, and continues without creating a duplicate.

Run the same sequence with failure before request delivery, during server processing, after commit, after local acknowledgment, and during reconciliation. Also test a tool that does not support idempotency. That tool should be classified as at-most-once with manual uncertainty handling, wrapped by a durable command service, or excluded from suspendable workflows. “Retry until success” is not a safe policy for external side effects.

Stop condition: Any duplicate irreversible effect, lost completed effect, or outcome that cannot be classified from durable evidence stops progression. The recovery path is to quarantine the actor, block further tool execution, compare local intent with the remote ledger, apply an approved compensation or manual decision, and create a regression case before unquarantining the workflow.

8. Stress Resume Storms and Cluster Capacity

Suspension can improve idle economics while making active demand more bursty. Common triggers include the start of a business day, a scheduled campaign, a webhook fanout, restoration after a regional dependency outage, or retries following a routing fault. Capacity must therefore be sized for simultaneous restoration and the downstream work that follows, not for the average number of active actors.

8.1 Model Concurrent Wake-Up and Node Pressure

Build a mass-wake model from the distribution of snapshot sizes, CPU and memory needed during restore, active task resources, WorkerPool free slots, node allocatable capacity, snapshot-store throughput, network bandwidth, token refresh, and downstream connection limits. Standard Kubernetes resource requests and limits remain the baseline for the substrate control workloads and workers: scheduler placement follows requests, limits are enforced differently for CPU and memory, and low average usage does not make overcommitted peak demand safe.

Run step tests at 10, 25, 50, 100, and 125 percent of the largest credible wake-up. Randomize snapshot size and task duration. Include a cold node pool, autoscaler delay, reduced snapshot throughput, and a downstream database with a strict connection ceiling. Measure queue age, free workers, restore bandwidth, CPU throttling, memory pressure, node provisioning, token errors, reconnects, and task completion.

Preserve headroom after the benchmark reaches steady load. A practical starting requirement is 30 percent free capacity in bottlenecks that cannot scale within the response objective, with no memory pressure, retry amplification, or downstream exhaustion. The percentage is a risk decision, but a test that passes only at saturation has no recovery margin.

Test backpressure explicitly. New resumes should queue with bounded age, shed or defer low-priority work, and protect already active sessions. A worker shortage must not cause all callers to retry at once. Use jittered client retries, a global concurrency budget, per-tenant fairness, and an overload response that can be distinguished from data loss. Cost the warm reserve required to meet this behavior, because it is part of the substrate economics.

8.2 Test Node Loss, Rescheduling, and Version Changes

Inject node loss at four boundaries: while an actor is active, while a snapshot is being created, after the snapshot is committed but before the worker is released, and during restoration. Use controlled drains and node termination in a dedicated non-production project. For storage faults, prefer permission denial, delayed reads, truncated test objects, or checksum mismatch in synthetic artifacts rather than modifying real tenant data. The expected result is either correct recovery from an accepted snapshot or an explicit quarantine, never silent continuation from uncertain state.

Version changes are a separate failure class. Record application revision and state schema in the application envelope. Test a compatible patch, a schema migration, a removed field, changed serialization, and an image that cannot interpret the old state. The resume gate must reject incompatible state before tool execution. If migration is supported, make it deterministic, idempotent, observable, and reversible, and preserve the pre-migration snapshot until the new state passes verification.

Do not mix substrate upgrades, node-image changes, application revisions, and state-schema migrations in one pilot step. Pin each ring, change one boundary at a time, and retain a known-good rollback pool. If a version change raises checksum mismatches, failed restores, or tail latency beyond gate, stop promotion, route new sessions to the control platform, and keep affected snapshots read-only for investigation.

Recovery is complete only when the restored agent passes state validation, external side effects reconcile, the user-visible task succeeds, capacity returns to normal, and evidence explains the failure. A rescheduled process with a green health check is not sufficient.

9. Compare Agent Substrate With Standard Pods and Agent Sandbox

The control groups answer different questions. Standard Pods show the mature Kubernetes baseline. GKE Agent Sandbox provides a managed add-on for isolated, stateful, single-replica environments, with warm pools, stable identity, Pod snapshots, and documented default-deny network behavior. Agent Substrate builds on sandbox concepts but decouples an Actor from a dedicated Pod or Worker so suspended state can live in storage and active actors can share a warm worker pool.

Run identical requests and compare successful business outcomes. Do not give Agent Substrate a smaller state set, warmer cache, looser isolation, or different downstream limits. Report idle density, active throughput, end-to-end latency, failure behavior, recovery time, and cost for each option. A specialized runtime belongs in the platform only when it wins on the workload that needs it without weakening the control plane around it.

Some estates will use more than one model. Standard Pods may remain appropriate for continuously active or strongly supported workloads; Agent Sandbox may fit isolated stateful sessions that do not need extreme suspended density; Agent Substrate may fit qualified, mostly idle actors after the guarded pilot. Uniformity is valuable, but forcing every agent into one runtime can be more expensive and less reliable than a small number of explicit service classes.

10. Measure Cost, Density, and Operational Complexity

The install documentation states that Agent Substrate has no extra product charge in GKE, but that does not make the runtime free. Include nodes and warm workers, Cloud Storage capacity and operations, network transfer, the substrate database, monitoring and log ingestion, egress controls, backups, support plan, engineering time, on-call load, security review, and incident recovery. Broader agentic AI cost governance is useful background; this benchmark must attribute the substrate-specific cost of suspension and restoration.

Use a unit that penalizes failure: total platform cost divided by the number of sessions that complete the business task and pass correctness gates. Also report cost per active minute, cost per resume, storage cost per suspended actor-day, and capacity-reserve cost. Do not divide by registered actors or attempted resumes, because those denominators reward density even when work is wrong or unavailable.

cost_per_successful_active_session =
  (compute + warm_capacity + snapshot_storage + storage_operations
   + network + database + telemetry + support + operations)
  / sessions_that_pass_correctness_and_isolation_gates

Compare at multiple idle ratios, snapshot sizes, wake-up rates, and retention periods. The break-even point may disappear when snapshots are large, sessions wake frequently, the workload needs a large warm reserve, or operator time dominates. Conversely, a long-lived assistant that is active for minutes and idle for days can produce a strong result. Publish the sensitivity curve rather than one favorable point.

Economic acceptance gate: The substrate must beat the control platform by the predeclared target at the same reliability and security level after all shared and incremental costs are allocated. If savings depend on excluding storm headroom, storage, observability, or operational labor, the gate has not passed. A pilot can be technically successful and still exit because the complexity-adjusted economics are weak.

11. Instrument State Drift, Failed Resumes, and Latency

Start with established observability foundations, then add signals for the state machine. The Google Cloud reliability pillar calls for scoping how components and actions flow, continuous observation, efficient response, and testing recovery from failures and data loss. That means the dashboard must explain whether a slow or failed request is waiting for a worker, reading a snapshot, restoring process state, refreshing identity, reconnecting, reconciling a tool, or executing the task.

Use non-sensitive correlation fields across router, actor, worker, application, and tool telemetry: run ID, tenant pseudonym, actor ID, session ID, request ID, state epoch, application revision, state-schema version, substrate version, worker pool, node, scenario, and failure-injection tag. Do not log prompts, snapshot contents, credentials, private tool responses, or full user identifiers merely to make debugging convenient.

Minimum metrics and events include:

• Suspend request count, suspend duration, snapshot bytes, snapshot commit failures, and time to release the worker.

• Resume queue age, worker claim time, snapshot read and restore time, reconciliation time, readiness time, and end-to-end task latency at percentiles.

• Checksum or file-manifest mismatch, state-epoch rollback, version rejection, failed resume, quarantine, and manual recovery.

• Network reconnect attempts, authentication refresh, token-exchange error, dependency failure, and connection-pool saturation.

• Tool operations by pending, confirmed, duplicated, compensated, and uncertain outcome.

• Active and suspended actors, worker occupancy, free workers, node pressure, autoscaler lag, snapshot throughput, and cost allocation.

The installer provides dashboards for routing latency, snapshot size, and gRPC traffic. Treat them as infrastructure evidence, not application proof. Add alerts that page immediately for any cross-tenant event, checksum corruption, or duplicate irreversible effect. Alert on tail-latency burn, growing queue age, repeated failed resumes, worker exhaustion, and downstream pressure before the user-visible SLO is consumed.

Retain benchmark evidence long enough to compare versions and investigate delayed reports, but apply the same data-minimization policy as production. Evidence should show the decision without embedding the sensitive state it is meant to protect.

12. Run a Guarded Non-Production Pilot

Keep the pilot in a dedicated non-production project and cluster until the support, correctness, isolation, and recovery gates have passed. Separate duties: the runtime team owns substrate deployment and worker capacity; the application team owns state and side effects; security owns threat tests and egress approval; SRE owns SLOs, telemetry, incident command, and rollback; finance or FinOps owns unit-cost validation; and the business owner accepts residual risk.

Every ring starts from a pinned substrate version, application image, ActorTemplate, node image, and benchmark corpus. Use a promotion change request that lists the new actor ceiling, expected wake-up profile, monitoring coverage, open risks, and rollback trigger. Do not combine a ring expansion with an upgrade or schema migration.

Schedule failure injection as part of the ring, not as an optional exercise after success. At minimum, drop the tool response after remote commit, expire credentials during suspension, deny snapshot access, drain a worker node, remove a downstream endpoint, exhaust a connection pool, introduce an incompatible state schema, and wake the planned peak population. Record whether the alert, incident owner, containment, recovery, and evidence all worked.

Run rollback before promotion. Route new sessions to the control platform, suspend or quarantine substrate actors, verify that durable workflow state remains available, reconcile outstanding side effects, and restore a representative session through the control path. Measure time to contain and time to recover. A rollback document that has not been executed is a hypothesis.

Hold a fixed evidence review. Record failed hypotheses, unresolved defects, support questions, cost variance, and temporary exceptions. Success means a defensible adoption or rejection decision, not an absence of reported failures.

13. Define Production Entry, Rollback, and Exit Criteria

Production entry requires a signed decision from platform, application, security, SRE, capacity, data governance, and business ownership. Limited-GA allowlist acceptance and a documented support path are mandatory but not sufficient. The application must also demonstrate that a restored actor is the correct actor, for the correct tenant, at a compatible revision, with reconciled external state and enough capacity to finish useful work.

Immediate stop conditions include: cross-tenant access, snapshot disclosure, wrong identity, unapproved egress, duplicate irreversible action, unrecoverable state corruption, silent version mismatch, inability to determine an external outcome, an unsupported production configuration, or a resume storm that threatens shared services. Pause new actor creation and new resumes where safe, preserve evidence, and move affected actors to quarantine rather than allowing automatic retries to multiply the incident.

Recovery path: establish the last durable state epoch, validate the most recent accepted snapshot, refresh identity, reconcile each uncertain side effect, restore through a known-good version or the control platform, execute a deterministic verification step, and obtain application-owner approval before normal tool execution resumes. If no trustworthy snapshot exists, rebuild from durable state and record the lost ephemeral context instead of guessing.

Rollback path:

1. Freeze promotion and route new sessions to standard Pods or the approved Agent Sandbox control path.

2. Stop or quarantine affected substrate actors; block tool execution until their external outcomes are reconciled.

3. Preserve the minimum incident evidence, then restore or reconstruct sessions from durable authoritative state.

4. Verify user-visible task completion, identity scope, and side-effect ledger on the control platform.

5. Drain workers, revoke unneeded IAM, remove runtime resources through the installation-specific cleanup procedure, and delete snapshots according to retention policy.

6. Close the change only after monitoring, billing, storage, credentials, and residual artifacts have been checked.

Exit the adoption effort, without treating the pilot as a failure, when qualified workloads are too active to benefit, snapshot sizes or retention make cost unattractive, version compatibility is operationally fragile, the egress model cannot satisfy policy, support scope is inadequate, or recovery labor exceeds the savings. Revisit only when the product status, application design, or workload economics materially change.

14. Decide Whether the Substrate Belongs in the Platform

GKE Agent Substrate belongs in a production platform only after four claims are proven together: suspension lowers the complete cost of qualified idle-heavy agents; restoration preserves the correct logical session; sandbox, identity, storage, and network controls prevent cross-tenant or unauthorized access; and the organization has an explicit supported operating and rollback path. A sub-second restore is useful evidence for one stage of the lifecycle, not a substitute for end-to-end correctness.

The safe outcome can be narrow adoption, continued evaluation, or rejection. A defined service class for reversible, idempotent, mostly idle workloads may be stronger than a broad default. Keep the benchmark, threat model, support record, and failure drills as living release gates as the platform changes.

Engineers building the transferable foundations behind this evaluation can review the Refonte Learning Agentic AI Engineer Program. Its published format is three months at approximately 12 to 14 hours per week, with coverage spanning agent frameworks, tool calling, retrieval-augmented generation, memory, API integration, evaluation, observability, deployment, and guardrails. Those foundations support disciplined runtime evaluation; the program is not presented here as training for Agent Substrate or an allowlisted GKE deployment.

Authorize production only when the evidence survives an adversarial restore, a mass wake-up, a node failure, a version change, and a rehearsed rollback. If any of those tests depends on hope, keep the workload on the control platform.