An EC2 Instance Is “Unmanaged” for Many Reasons: Build a Systems Manager Recovery Runbook

On September 9, 2026, AWS expanded Systems Manager diagnosis for unmanaged EC2 instances beyond network configuration to include IAM permissions, SSM Agent version, instance status checks, operating-system configuration, Systems Manager configuration such as Default Host Management Configuration, and other registration failures. That is useful, but a diagnosis result is not permission to change production. An instance can also be shown as unmanaged for an expected reason, most notably because it is stopped and the agent has no active connection.

The operational risk is misclassification. Teams often reinstall SSM Agent because the console says Not managed, even when the failure is a network ACL, an endpoint security group, a missing instance profile, a failed EC2 status check, disk exhaustion, or a DHMC prerequisite. That reflex destroys evidence and can add outage time without restoring the management channel. The broader discipline described in modern system administration responsibilities applies here: establish state, ownership, dependency, and reversibility before touching the guest.

This runbook starts with inventory and safe observation, then moves through expected disconnection, coverage, network, identity, agent, instance health, and OS branches. It distinguishes diagnosis from remediation, management connectivity from application health, and a successful Automation execution from a recovered service. Use it to authorize changes, define stop conditions, preserve rollback, validate node identity, and measure whether the fleet remains manageable after the incident is closed.

1. Define “Unmanaged” Before Treating It as an Incident

In Systems Manager, a managed node is not merely an EC2 instance that exists or has SSM Agent installed. The node must satisfy the platform prerequisites, have a running agent, possess an authorized identity path, and reach the Systems Manager service endpoints in the correct account and Region. The managed-node availability guidance treats operating-system support, agent state, IAM permissions, and endpoint connectivity as separate requirements. Failure in any one can remove the active management relationship.

An unmanaged label is therefore an observation, not a root cause. It can describe a stopped instance, a running instance that never completed onboarding, a previously managed node whose heartbeat disappeared, or a node whose inventory view is stale. Systems Manager node insights are scan-based and time-bound; the count is evidence about a particular scan, not proof that every account, Region, or instance lifecycle state has been examined.

The most important exception is documented in node insight behavior for stopped instances. A stopped EC2 instance appears unmanaged because SSM Agent has no active connection. When the instance starts, it should normally reconnect and appear managed within about five minutes. Do not open an incident for the expected stopped state unless the business policy says the instance should be running, or it remains unmanaged after startup and the reconnection window.

Decision tree: begin with instance state and last managed heartbeat, then follow only the branch supported by evidence.

1.        Is the instance stopped, stopping, pending, shutting down, or terminating? If stopped or stopping is expected, record the state and owner; do not remediate SSM configuration.

2.        If the instance is running, when was its last managed heartbeat or last successful Systems Manager operation? Separate “never managed” from “previously managed and lost.”

3.        Do EC2 system, instance, and attached-EBS status checks pass? A failed health check takes precedence over agent repair.

4.        Was the intended account and Region actually included in the diagnosis execution? Absence from a console summary is not proof of health.

5.        Classify the strongest supported cause: expected disconnected state, control-plane setup, network path, IAM identity, DHMC, agent, instance health, OS, or unidentified.

6.        Choose a cause-specific action with an owner, approved blast radius, recovery test, and rollback. Reinstallation is only the agent branch.

Acceptance gate: close the classification step only when the instance state, account, Region, last contact, current status checks, intended identity path, and responsible owner are recorded. “Unmanaged” by itself is not a diagnosis.

2. Build a Fleet Inventory Across Accounts and Regions

Start with the estate you intend to govern, not the nodes currently visible in one console. The unmanaged-instance diagnosis overview notes that impacted counts can include instances that were Running, Stopped, or Stopping when the scan occurred. Treat that result as a static snapshot. Build an inventory that can reconcile EC2 inventory, Systems Manager node state, AWS Organizations scope, and the account-Region pairs the diagnosis actually covered.

2.1 Separate Running, Stopped, and Transitional Instances

Partition the fleet by EC2 lifecycle state before calculating an unmanaged rate. Stopped instances belong in an expected-disconnected population unless policy requires them to be running. Pending and stopping instances need a time-bounded observation window because both the guest and management channel can be legitimately unavailable. Terminating instances usually belong to lifecycle cleanup, not onboarding remediation.

For running instances, add two more dimensions: whether they have ever been managed and whether they recently changed state, AMI, subnet, profile, or launch-template version. A node that never registered suggests an image, IAM, DHMC, or initial network-path defect. A node that was managed until a known deployment suggests configuration drift, service failure, resource exhaustion, or a shared infrastructure change. That distinction should drive both the evidence request and the rollback candidate.

Stop condition: do not start a fleet-wide repair when the apparent spike is explained by an approved stop schedule, an Auto Scaling replacement wave, a Region that was not scanned, or a stale scan. First normalize the denominator to the intended running population.

2.2 Record Last Contact, Ownership, and Business Criticality

A last-contact timestamp converts an ambiguous label into an incident timeline. Capture the last healthy Systems Manager contact, the first failed observation, the last deployment or configuration change, and the first human action. If no contact has ever occurred, record the launch time and expected onboarding deadline. If the node has been disconnected for a long period, verify that inventory retention has not hidden it from the view you are using.

Ownership and business criticality decide what you may do. A disposable stateless worker in an Auto Scaling group may be safer to replace from a corrected launch template. A single-instance database helper with local state may require an EBS snapshot, application quiescence, and an approved outage before restart. A shared subnet or VPC endpoint finding can affect many services, so the network owner must approve even if only one instance first exposed the problem.

•         Criticality: customer-facing, internal production, regulated, development, or disposable.

•         Recovery model: replace, reboot, stop/start, restore volume, or repair in place.

•         State risk: EBS only, instance store present, in-memory queues, local files, or unflushed application data.

•         Authority: who may change security groups, NACLs, endpoint policies, IAM roles, DHMC, agent packages, and the guest OS.

•         Validation owner: who confirms application behavior after the management channel returns.

3. Verify Diagnosis Coverage, Permissions, and Execution History

The unified console can operate at different scopes. The Systems Manager diagnose-and-remediate model allows a delegated administrator to select targets across an organization’s accounts and Regions, while single-account operation is scoped one Region at a time. Before trusting a fleet result, compare the intended scope with the actual account and Region selection, organizational units, excluded accounts, enabled Regions, delegated-administrator status, and setup failures.

A diagnosis can succeed technically while omitting the place where the incident exists. Build a coverage ledger with one row per intended account-Region pair and record setup status, scan execution ID, start and end time, result, and node count. Mark not scanned, not onboarded, access denied, timed out, and zero instances as different states. A green execution in the home Region is not proof that every member account and opt-in Region participated.

Use diagnosis execution history as the audit trail, not a screenshot of the summary card. The history distinguishes Diagnosis and Remediation executions, records start and end times and status, and exposes step inputs and outputs. Export or capture the execution identifiers and failed steps before launching another run; repeated executions can otherwise bury the evidence that explains why coverage was incomplete.

Coverage gate: the numerator is account-Region pairs with a completed diagnosis execution in the incident window. The denominator is the approved account-Region inventory, not only Regions that returned data. Report exclusions explicitly.

Authorization gate: the operator must be able to read the runbook preview, target scope, and execution details. The remediation role must be separate from broad human administrator access where practical, and its permissions must match the approved action rather than the entire service.

4. Classify the Root Cause Before Remediation

Classify the cause from the strongest available evidence before choosing a fix. The current diagnosis workflow groups findings into network configuration, IAM, SSM Agent version, instance status, OS configuration, Systems Manager service configuration such as DHMC, hybrid activation, and unidentified issues. For EC2, use those categories as investigation queues, not mutually exclusive truths; a node can have an old agent and a blocked endpoint at the same time.

4.1 Network, Endpoint, and Network ACL Failures

Network findings are the branch for which the documented workflow can present manual guidance or an Automation runbook preview. The diagnosable network categories include instance and endpoint security groups, VPC DNS attributes, missing or incorrectly associated interface endpoints, endpoint policies, and network ACL behavior. That does not make every network finding automatically remediable. A shared NACL, custom DNS path, firewall appliance, or cross-VPC design often requires a network owner and manual change.

Network ACLs are stateless. For an instance-initiated HTTPS session, the relevant path must permit port 443 in the outbound direction and return traffic on ephemeral ports, documented for this diagnosis as 1024 through 65535, in the opposite direction. Check both the instance subnet and endpoint subnet, rule order, protocol, CIDR, and deny rules. Do not solve an ACL finding by opening all ports to the internet; restore the narrow path between the node population and approved endpoint or egress destination.

Evidence standard: capture the effective security groups, NACL entries with rule numbers, route-table associations, endpoint IDs and subnets, endpoint policy, VPC DNS attributes, resolver path, proxy variables, and the timestamp of the last known-good connection. A reachability theory without configuration evidence is not a network diagnosis.

4.2 IAM, DHMC, Agent, Status, and OS Failures

The remaining categories mostly direct the operator to guidance and manual investigation. As of September 14, 2026, the official unmanaged-EC2 workflow exposes runbook selection for network findings, while IAM, agent-version, status-check, OS, service-configuration, and unidentified findings use Learn more paths. An identified cause is therefore not the same as an automatically remediable cause.

Keep identity paths separate. An instance profile is attached to a specific EC2 instance, while DHMC is an account-and-Region service setting that can authorize eligible instances without a per-instance profile. Changing both during the same incident makes the recovery non-attributable. Determine which path was intended, which path the agent is actually using, and whether the prerequisite agent and metadata configuration support it.

Stop condition: if two cause classes remain equally plausible and the next step is mutating, pause. Gather a discriminating signal or use an isolated reproduction. Do not stack fixes until something turns green; that destroys causal evidence and makes rollback ambiguous.

5. Diagnose Network Paths to Systems Manager Services

SSM Agent initiates the connection to Systems Manager, so the instance does not need an inbound security-group rule for the management channel. The SSM Agent network troubleshooting guidance requires outbound HTTPS connectivity to the Systems Manager and message endpoints. For current designs, validate ssm.region.amazonaws.com and ssmmessages.region.amazonaws.com; consult the Region-specific guidance before depending on ec2messages, because support differs between older and Regions launched from 2024 onward.

Private instances can use NAT or Systems Manager interface VPC endpoints. For PrivateLink, verify interface endpoints in the correct VPC and Region, private DNS where expected, subnet placement, route and resolver behavior, and an endpoint security group that permits inbound TCP 443 from the managed-instance subnets or approved security groups. The instance security group must permit the corresponding outbound path. Endpoint policies can also block otherwise reachable traffic, so a successful TCP handshake is not proof of authorization.

Run the investigation from the outside in. First compare a healthy peer in the same subnet. Then verify VPC DNS attributes, hostname resolution, route table, security groups, endpoint associations, NACLs, proxy configuration, and TLS trust. Use VPC Flow Logs or firewall logs when available, but interpret them carefully: ACCEPT does not prove DNS, TLS, endpoint policy, IAM, or application-layer success, and REJECT must be tied to the correct interface and five-tuple.

Worked example and failure injection: in a non-production subnet that mirrors production, remove the return-path ephemeral-port allow rule from the test NACL under an approved experiment. The running test instance should lose the management channel while EC2 status checks remain healthy. Confirm the diagnosis identifies the network path, capture the failed flow evidence, restore the exact NACL rule and number, and verify reconnection. Never inject this fault in a shared production subnet.

For a real incident, suppose three payment workers in one subnet became unmanaged at 02:14 after a network-policy deployment. Other subnets remain healthy, system and instance checks pass, and all three share the same NACL. The evidence supports a shared network branch, not three agent reinstalls. Capture the current and previous NACL, obtain the network owner’s approval, restore only the missing return rule, and watch the nodes reconnect. If contact does not return, roll back the change and continue with endpoint, DNS, proxy, IAM, and agent evidence rather than widening the ACL.

Acceptance gate: the original endpoint path is reachable through the approved route, DNS returns the expected private or public destination, no unapproved broad egress was added, affected nodes reconnect within the defined window, and healthy peer traffic remains stable. Application checks are still required separately.

6. Diagnose IAM Roles and Default Host Management Configuration

For instance-profile onboarding, inspect the EC2 association, not just the IAM role’s existence. The role must be attached through an instance profile, trusted for EC2, and grant the Systems Manager permissions required by the organization. AWS recommends AmazonSSMManagedInstanceCore or an equivalent custom policy in the instance permission guidance, but least privilege may require additional controls for S3, KMS, logs, or application-specific operations. Do not attach an administrator policy to prove a theory.

For DHMC, verify the account and Region, selected management role, and prerequisites in the Default Host Management Configuration documentation. Eligible instances must use IMDSv2 and SSM Agent version 3.2.582.0 or later. Activation is regional and can take time to affect instances, so a newly enabled setting is not evidence that every node has already registered. Capture the service setting and role ARN before changing either.

Precedence matters. If the instance profile grants ssm:UpdateInstanceInformation, SSM Agent uses the instance-profile permission path rather than DHMC. A team can therefore enable DHMC correctly and still observe the instance behaving according to an old or broken profile. Decide whether the design standard is per-instance profile or DHMC, then repair that path only. This is a concrete application of zero-trust access controls: the goal is a verified, bounded identity, not simply more permissions.

Do not delete registration state as a routine fix. DHMC relies on local SSM registration information under /var/lib/amazon/ssm on Linux and C:\ProgramData\Amazon on Windows. Removing those files can prevent DHMC credentials from being acquired and can force a different recovery path. Preserve the directory, permissions, timestamps, and relevant log evidence before any reset considered by AWS Support or an approved runbook.

1.        Confirm the account, Region, instance ID, profile association ID, role ARN, and policy versions.

2.        Confirm IMDSv2 configuration and that the guest can obtain its intended role credentials without exposing credential material.

3.        Read the DHMC service setting and selected role in the affected Region.

4.        Determine whether ssm:UpdateInstanceInformation is present in the instance-profile path and therefore takes precedence.

5.        Make one identity change, observe CloudTrail and node registration, then either accept or restore the previous state.

Rollback: reattach the original instance profile or restore the prior DHMC setting and role. Confirm that the rollback does not remove permissions used by other instances, and record any temporary policy version so it can be deleted after evidence is preserved.

7. Diagnose SSM Agent Installation, Version, and Service State

Reinstallation is appropriate only after the agent branch is supported. First establish whether the package exists, whether the service unit matches the installation method, whether the process is running, and whether logs show endpoint, credential, certificate, proxy, permission, disk, or registration errors. The official service-status commands differ for standard Linux packages, Ubuntu Snap installations, and Windows Server, so using the wrong service name can create a false “agent missing” conclusion.

# Amazon Linux, RHEL, Debian, and similar systemd installations
sudo systemctl status amazon-ssm-agent
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

# Ubuntu Snap installation
sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent.service
sudo snap start amazon-ssm-agent

# Windows PowerShell
Get-Service AmazonSSMAgent
Start-Service AmazonSSMAgent

Read logs before restart when possible. On Linux, the primary files are /var/log/amazon/ssm/amazon-ssm-agent.log and /var/log/amazon/ssm/errors.log; on Windows they are under C:\ProgramData\Amazon\SSM\Logs. Capture the relevant interval, service-manager output, package history, free disk and inode capacity, proxy variables, system time, and recent certificate changes. Redact credentials, tokens, commands, and sensitive output before attaching evidence to a ticket.

Check the installed version using the SSM Agent version procedures, then compare it with the minimum required by the feature you intend to use and the organization’s security baseline. “Latest” and “installed” are different facts, and a package repository may lag the release channel. DHMC requires at least 3.2.582.0, while the diagnostic utility described below requires 3.1.501.0. Keep version policy in the image pipeline and update association rather than discovering it during every incident.

# Ubuntu Snap
sudo snap list amazon-ssm-agent

# Windows PowerShell
& "C:\Program Files\Amazon\SSM\amazon-ssm-agent.exe" -version

# Availability diagnostics, from the SSM Agent installation directory
ssm-cli get-diagnostics --output table

Starting with Agent 3.1.501.0, ssm-cli diagnostics can test primary management prerequisites from the node. Treat its output as a structured clue, not an application-health test. If it reports an endpoint or credential failure, resolve that cause first. If the service will not start, inspect the exact error, ownership, executable, configuration, and dependency state before replacing the package.

Reinstall only when package files are missing, corrupt, unsupported, or an approved vendor procedure requires it. Capture the package source, checksum or repository, current version, configuration, registration directory, service definition, and logs. Reinstall the same approved version first when the goal is repair; upgrade as a separate controlled change unless the diagnosed issue is specifically version incompatibility. A clean install that still cannot reach endpoints or obtain credentials confirms that reinstallation was not the root-cause fix.

Recovery gate: the service is enabled and stable, diagnostics pass the intended identity and endpoint checks, the node appears with the correct account, Region, and EC2 instance ID, and a read-only management action succeeds. Continue to application validation before closure.

8. Diagnose Instance Health and Guest Operating-System Failures

The OS branch also includes less dramatic failures: a full root filesystem, exhausted inodes, read-only remount, invalid proxy, expired private CA, clock drift, damaged package database, blocked local firewall, or startup job that prevents the agent from launching. Collect evidence through approved direct access, console output, serial access where supported, or offline volume inspection. Do not weaken host security merely to regain SSM; use the narrowest break-glass path and remove it after recovery.

8.1 Use Console Output, Status Checks, and Disk Evidence

Amazon EC2 runs system, instance, and attached-EBS status checks automatically. System checks cover the AWS host and underlying infrastructure. Instance checks cover guest networking and software conditions such as startup configuration, memory exhaustion, filesystem corruption, or an incompatible kernel. Attached-EBS checks cover volume reachability and I/O on supported instances. These signals determine whether to wait, reboot, stop/start, fail over, or repair the guest.

aws ec2 describe-instance-status \
  --instance-ids i-0123456789abcdef0 \
  --include-all-instances \
  --region us-east-1

aws ssm describe-instance-associations-status \
  --instance-id i-0123456789abcdef0 \
  --region us-east-1

For disk-related suspicion, record filesystem capacity, inode use, mount state, EBS burst or latency symptoms, and whether the agent can create files in its log and data directories. If the application writes heavily, coordinate quiescence and data consistency before snapshots or offline repair. A management-channel recovery that leaves a corrupt filesystem, stalled queue, or unavailable application is a false closure.

Stop condition: do not reboot or stop/start a stateful instance until the owner has assessed in-memory state, instance-store data, public-IP change, cluster membership, failover readiness, and the recovery objective. A status-check failure is evidence for escalation, not blanket permission to cycle the node.

8.2 Escalate to EC2Rescue or Direct Recovery Safely

When the guest is unreachable and evidence supports OS-level repair, use an approved offline method. The EC2Rescue Automation process can create temporary infrastructure and a helper instance, stop the affected instance, back it up, attach its root volume to the helper, run repair actions, reattach the volume, and restart the original instance. That is a high-impact recovery workflow, not a diagnostic read-only step.

Review the AWSSupport-ExecuteEC2Rescue runbook reference before execution. The documented runbook stops the instance, creates an AMI backup, can change a public IP when no Elastic IP is used, and loses instance-store data on stop. The current runbook does not support encrypted root volumes. Confirm the instance type, subnet and endpoint reachability for the helper, IAM permissions, logging destination, KMS implications, and application outage approval.

Direct recovery may instead mean detaching a root volume after stop, attaching it to a controlled helper in the same Availability Zone, mounting it without unintended writes, repairing configuration, and restoring the original attachment. Require two-person review for device mapping and volume identity. Preserve the original volume or snapshot until the repaired instance has passed both management and application gates. Never delete the known-good fallback because the agent appears online once.

Recovery path: restore boot and filesystem health, reattach the correct root volume, start the instance, confirm all EC2 checks, then validate SSM identity and application behavior. If any step is ambiguous, stop and return to the backup AMI or original volume rather than layering more changes.

Rollback path: stop the repaired instance through the approved process, restore the original volume or launch from the backup AMI, reapply the captured profile and network associations, then revalidate. Record data created after the backup so the application owner can reconcile it.

9. Choose Automatic or Manual Remediation by Risk

Automatic remediation is a delivery mechanism, not a risk rating. The runbook preview classifies actions as mutating, non-mutating, or undetermined. The remediation impact definitions explain that undetermined steps can invoke Automation, Run Command, Lambda, Step Functions, scripts, webhooks, or other orchestration whose final effect is not known from the preview. Treat undetermined as requiring investigation, not as a midpoint between safe and risky.

For unmanaged EC2 diagnosis, current automatic remediation is primarily associated with supported network findings. Even there, preview the target accounts, Regions, OUs, resource count, and each step. Decide whether the change touches an instance-specific security group or a shared endpoint, route, DNS setting, or NACL. A one-click operation that modifies a shared control can have a larger blast radius than a manual change to one node.

Use four authorization rings. Ring 0 is diagnosis only. Ring 1 covers a non-production, single-resource remediation with full rollback. Ring 2 covers a production canary in one account and Region during an approved window. Ring 3 covers a bounded fleet rollout after recurrence and application evidence remain clean. The principles in cloud security best practices apply directly: minimize permissions, narrow targets, retain evidence, and separate remediation authority from validation.

1.        Preview and record every mutating and undetermined step, target, account, Region, role, and expected API effect.

2.        Capture configuration and recovery state before execution. Do not rely on the runbook to preserve the rollback you need.

3.        Run one canary whose failure is observable and recoverable without customer impact.

4.        Watch management-channel, network, IAM, EC2, and application signals through at least one normal workload interval.

5.        Promote only when the canary passes the acceptance gate and no unrelated nodes regress.

Measurable acceptance gate: at least 99 percent of approved canary targets return to Managed within the defined five-minute reconnection window after the cause is removed; all have the correct identity; no new unmanaged nodes appear in the shared scope; and the application owner confirms its health checks, transactions, and backlog. Set a stricter target for critical fleets.

Stop condition: stop the execution or rollout when target scope differs from approval, the preview contains an unexplained undetermined action, rollback evidence is missing, a shared dependency changes unexpectedly, status checks deteriorate, or application error and latency thresholds are crossed.

10. Preserve Access, Snapshots, and Rollback Before Changes

Pre-remediation capture must match the change. For network work, save security-group rules, endpoint configuration and policy, route associations, NACL entries and numbers, VPC DNS attributes, resolver rules, proxy configuration, and flow-log queries. For IAM, save the profile association, role trust, attached and inline policies, policy versions, DHMC setting, and intended precedence. For agent and OS work, save service definitions, package version, configuration, logs, registration data, disk state, console output, and recent package history.

For changes that can alter the guest or storage, create the appropriate backup. Amazon EBS snapshot guidance describes point-in-time volume backups and notes that cached application or OS data is not included unless it has been written. Coordinate application consistency, write quiescence, and multi-volume requirements. A snapshot in pending state is not the same as a tested restore, and a snapshot does not preserve security groups, IAM associations, DNS, routes, or instance-store data.

Preserve a second access path before changing the primary one. That may be an approved break-glass console path, EC2 Serial Console where supported, a tested bastion route, a replacement instance from a launch template, or an offline-volume recovery plan. Do not create a permanent inbound administrative rule as an emergency shortcut. Time-box temporary access, restrict source and identity, log its use, and remove it after SSM is stable.

Rollback decision: define it before the change with an owner and time threshold. “We can probably rebuild it” is not rollback for a stateful or regulated system.

11. Restore Management Connectivity and Validate Node Identity

Restoration has two gates. The management-channel gate proves that the correct node is connected to the correct Systems Manager account and Region under the intended identity. The application gate proves that the workload is healthy. Do not merge them. A green agent can coexist with failed application dependencies, stale local state, partial boot, or a node that should have been replaced rather than repaired.

After remediation, allow the documented reconnection interval, then compare instance ID, account, Region, hostname, private IP, platform, tags, and owner record. Confirm the node is Managed, the expected profile or DHMC path is in use, agent version and service state are stable, and there is no duplicate or stale inventory record. Run one read-only action such as hostname and time retrieval through the approved command document, with output logging and narrow targeting.

Then ask the application owner to validate service-specific evidence: load balancer health, process and dependency status, queue depth, database connectivity, scheduled work, error rate, latency, transaction or synthetic result, and data consistency. Restarting SSM Agent or completing an Automation execution proves only that the action ran. It does not prove customer traffic, batch completion, or business data integrity.

•         Management state remains healthy through two heartbeat intervals and one representative management action.

•         Identity fields match the EC2 inventory and intended authorization path.

•         EC2 system, instance, and attached-EBS checks pass.

•        Application owner confirms normal service and reconciles any ambiguous work during the outage.

•         Temporary access, policies, routes, or diagnostic artifacts are removed or assigned a dated owner.

False-closure rule: reopen the incident when the node disconnects again within the recurrence window, application health remains degraded, the restored identity is broader than approved, or scan coverage cannot prove that peer nodes are unaffected.

12. Schedule Recurring Scans and Prevent Regression

Systems Manager can run recurring unmanaged-instance diagnosis daily or weekly. Choose frequency from business criticality and fleet change rate, not convenience. Record the configured local start time, account and organization scope, enabled Regions, owner, alert path, and retention of execution results. A schedule that silently fails is not a control, so monitor the diagnosis execution itself.

Use recurring diagnosis as detection, then prevent recurrence at the source. Image pipelines should install and verify the approved agent, enable the correct service, set proxy and CA trust intentionally, and test registration in each target network pattern. Launch templates and account baselines should define the intended profile or DHMC model, IMDSv2, endpoint path, DNS, logging, and tags. The practices in system administration automation help make those checks repeatable, but remediation still needs evidence and bounded authority.

Move successful emergency fixes into infrastructure-as-code controls after review. Codify endpoint security groups, policies, NACLs, routes, IAM roles, profile attachments, DHMC setup, update associations, alarms, and inventory tags. Test drift detection against an isolated account or VPC before broad enforcement. Do not let an automation loop repeatedly reapply a fix to a node whose actual problem is disk failure or an unsupported image.

•         Image gate: boot a fresh instance, confirm status checks, agent service, version, identity, endpoint reachability, and Managed state.

•         Network gate: test public-egress and PrivateLink patterns, DNS, endpoint policy, and NACL return traffic.

•         Identity gate: prove profile and DHMC paths independently and document precedence.

•         Lifecycle and scan gate: reconcile stopped schedules, Auto Scaling replacement, and terminated inventory, then alert when an expected account-Region execution is missing rather than only when the unmanaged count rises.

13. Measure Fleet Coverage, Recurrence, and Recovery Time

Fleet metrics must expose blind spots and false recovery. Build them from execution history, EC2 inventory, node state, configuration changes, and incident records. The general principles of observability and operational signals apply: every indicator needs a defined population, timestamp, owner, threshold, and response. A single “managed percentage” hides stopped nodes, unscanned Regions, stale data, and repeated failures.

For automatic remediation, count only targets that pass management and application gates. A runbook status of Success is an execution result, not a recovered-node result. For recurrence, choose a policy window long enough to include normal patching, stop/start, scaling, and workload cycles. Review outliers and false closures in a monthly control meeting, then assign prevention work to the image, network, IAM, OS, or application owner.

Service objective example: 99.5 percent of intended running production instances are Managed, 100 percent of in-scope account-Region pairs have a successful daily or weekly diagnosis as designed, P95 unmanaged duration is below the approved threshold, and false closure remains below 1 percent. Set values from business risk and historical performance, not from this example.

14. Close the Incident With Evidence and Ownership

Close only when the expected state or root cause is documented, the approved remediation is attributable, rollback remains available for its retention period, and both management and application gates pass. The incident record should contain the instance and fleet scope, account and Region coverage, timeline, diagnosis and Automation execution IDs, configuration captures, logs, approval, change, observed effect, rollback decision, application validation, temporary-access cleanup, and prevention owner. Redact secrets and sensitive payloads without removing the evidence needed to reproduce the decision.

The durable acceptance standard is not “SSM Agent is online.” It is: the right node is manageable through the intended least-privilege path; EC2 and guest health are stable; the application is functioning; peer nodes were not harmed; recurrence is monitored; and the responsible team can reverse or rebuild the change. If the cause remains unidentified, close the outage only if service is restored, then keep a problem record open with bounded risk and an engineering owner.

Teams that need stronger transferable foundations in installation and configuration, networking, security, troubleshooting, virtualization, backup and recovery, command-line administration, and cloud systems can review the Refonte Learning System Administration Program. Its published format is six months at approximately 10 to 12 hours per week. Those foundations support disciplined fleet diagnosis, but the production controls in this runbook still require organization-specific architecture, access policy, change governance, and recovery testing.