AWS Daily Intelligence #29 - Unmanaged is a symptom, and now it names the cause

Executive summary

On 9 September AWS expanded what Systems Manager can tell you about an unmanaged EC2 instance, adding six additional categories of issues on top of the network connectivity it already diagnosed: IAM permissions, SSM Agent version, instance status checks, operating system configuration, Default Host Management Configuration, and hybrid activation.

This matters more than a diagnostics improvement usually would, because of what "unmanaged" costs. An instance Systems Manager cannot treat as a managed node cannot be patched, cannot be reached with Run Command or Session Manager, and does not appear in inventory. It is not merely unmonitored — it is absent from the tooling that proves your fleet is compliant, and it is absent quietly.

That is the real problem being addressed. "Unmanaged" is a single symptom sitting on top of at least ten distinct causes spread across networking, IAM, the operating system and the agent itself. Diagnosing it by hand means checking each in turn on a machine you cannot log into, which is why these instances accumulate. The diagnosis now returns a specific, actionable cause across most of that surface rather than an unidentified result.

What changed

The diagnosis previously covered the network path. It now covers most of the rest of the reasons an agent fails to check in.

Category	Examples of what it finds	Auto-remediable
Security groups and HTTPS	Instance outbound 443 blocked; ssm, ssmmessages or ec2messages endpoint security group blocking inbound 443	Usually
DNS configuration	DNS support or DNS hostnames disabled on the VPC	Yes
VPC endpoints	Missing PrivateLink endpoints; subnet not attached to an existing endpoint	Partly — the "not attached" findings have no runbook
Network ACLs	Outbound 443 blocked; inbound ephemeral ports 1024–65535 blocked	No — guidance only
IAM roles and permissions	No instance profile; missing AmazonSSMManagedInstanceCore; insufficient actions	Guidance
SSM Agent version	Agent older than the minimum supported version; agent lacking required protocols	Guidance
Instance status checks	System status check failure; instance status check failure	Guidance
Operating system	Agent service stopped; kernel panic or filesystem corruption; host firewall blocking 443; disk full	No — requires OS access
Default Host Management Configuration	DHMC not enabled; its role missing policies; its role's trust policy wrong	Guidance
Hybrid activation	Activation expired; registration limit reached; registration conflict; endpoint unreachable	Guidance

The six in bold are new. It is available in all Regions enabled by default, and it runs as Automation runbooks, so you pay standard Automation usage charges for what you run.

Architecture

Diagram: an unmanaged EC2 instance as one symptom with many causes, grouped into what the Systems Manager diagnosis covered before and what it covers now. The previously covered network family holds four categories: security group and HTTPS configuration, DNS and DNS hostname configuration, VPC endpoint configuration, and network ACL configuration. The six newly added categories are IAM roles and permissions, SSM Agent version, instance status checks, operating system configuration, Default Host Management Configuration, and hybrid activation. A second panel separates findings that come with an Automation remediation runbook from findings that come with written guidance only, noting that operating system issues are diagnosed by analysing instance console output and explicitly cannot be auto-remediated because they require direct access to the operating system, and that some VPC endpoint and network ACL findings also have no runbook. — One symptom, ten categories of cause. The diagnosis used to cover the leftmost family; it now covers most of the rest — though covering is not the same as fixing.

Two design details are worth knowing before you run it.

It works per VPC, not per instance. AWS states the diagnosis process examines each group of EC2 instances at once according to the virtual private cloud they belong to. That fits the failure mode: most causes in the network family are properties of a VPC or subnet, so one misconfigured endpoint association explains every unmanaged instance in that subnet at once. Expect findings that describe a group rather than a machine.

Some findings are read from the console output. The operating system category is diagnosed by analysing the instance console output, which is how Systems Manager can tell you a machine has a kernel panic or a full root volume without being able to reach the agent on it. It is also why AWS says automatic remediation is unavailable for that whole category: fixing it requires direct access to the instance operating system, which is exactly the access you have lost.

That produces a satisfying loop. The agent is down because the disk is full; the disk being full is why you cannot use the agent to clear it. Systems Manager can now at least tell you that is the situation instead of reporting an unidentified failure.

Business value

Patch compliance reporting is only as honest as the denominator. An unmanaged instance is missing from the report rather than failing in it, so a fleet with a hundred unmanaged machines can show 100% patch compliance across the ninety it can see. Closing that gap is the value, and it is a governance value more than an operational one.

The second value is triage time. The IAM, agent-version and status-check categories in particular replace a manual checklist that requires several consoles and a person who knows the order to check them in. Returning a named cause turns that into a ticket someone junior can action.

Security considerations

The IAM category is the one to read carefully, because it names the minimum permission set precisely: ssm:UpdateInstanceInformation, ssmmessages:CreateControlChannel, ssmmessages:CreateDataChannel and ec2messages:GetMessages. That is useful in both directions. It tells you what to grant, and it tells you what a least-privilege replacement for AmazonSSMManagedInstanceCore would have to contain if you are trying to avoid the managed policy.

Default Host Management Configuration deserves a mention on the same grounds. DHMC lets Systems Manager manage instances without an instance profile on each one, which is a real reduction in per-instance credential surface — and the diagnosis now catches the three ways it silently fails, including a role whose trust policy does not let Systems Manager assume it. A misconfigured DHMC role means every instance without an instance profile is unmanaged, and nothing about that presents as an error.

One caution: onboarding to the unified console requires creating IAM roles and managed policies for these operations, and for an organization a delegated administrator can then target across all accounts and Regions. That is appropriate for the job and worth reviewing as a piece of cross-account reach.

Cost considerations

Not free, and the pricing shape is easy to miss: the capability runs as Automation runbooks and you pay standard Automation usage charges for the runbooks you run. A one-off diagnosis across a large estate is a large number of runbook executions.

This is a small cost against the alternative — an engineer working through categories by hand — but it is a real one, and it argues for running diagnosis deliberately rather than on a schedule across everything. Diagnose when the unmanaged count moves, not every night.

Operational considerations

Expect a mixed result set. Several categories return guidance rather than a runbook, and two of the most common — network ACLs, and subnets not attached to an existing endpoint — are diagnose-only today. Planning as though the button fixes everything will produce a list of findings nobody owns.

There is also a prerequisite that will stop some teams before they start: Diagnose and remediate requires onboarding to the unified Systems Manager console first, including creating the IAM roles and managed policies the operations need. That is a change with its own approval path in most organisations, and it is worth starting before the day you need the diagnosis.

Tradeoffs

Approach	Works well when	Breaks down when
Run the expanded diagnosis	You have a standing unmanaged population and no idea why. It names causes across ten categories in one operation.	You expected it to fix them. Much of the new coverage is guidance, not remediation.
Prevent with a baseline	New accounts and new instances. DHMC plus endpoint and security group standards stops most causes existing.	The existing fleet. A baseline applies forward and does nothing for what is already adrift.
Diagnose by hand	One instance, one engineer, a known-good comparison nearby.	Any fleet. The categories span four consoles and the OS you cannot reach.

Implementation guidance

Get the count first. The number of EC2 instances in the account that Systems Manager does not treat as managed nodes is the metric this feature exists to reduce, and most teams have never put a number on it.

Then onboard to the unified console, accepting the IAM role and managed policy creation, and run a diagnosis against one VPC rather than the estate. Because findings are grouped by VPC, a single VPC is enough to reveal whether your problem is one systemic misconfiguration or a long tail of individual machines — and those two shapes need completely different responses.

Fix the systemic ones with a baseline change so they stop recurring, and treat the long tail as tickets. Re-run the diagnosis after the baseline change rather than before, so the second number means something.

Best practices

Track unmanaged node count as a compliance metric in its own right. Patch reports do not include what they cannot see.
Enable Default Host Management Configuration, and check its role's trust policy — a broken DHMC role makes every profile-less instance unmanaged, silently.
Run diagnosis per VPC first. The grouping tells you whether you have one problem or many.
Budget for it. It executes Automation runbooks and is billed accordingly.
Do not wait for the outage to onboard the unified console. The prerequisite has an approval path; the incident will not.

Who should adopt this

Anyone running a fleet large enough that "why is that one unmanaged" has ever gone unanswered, and anyone whose patch compliance is reported to an auditor. The second group should care most, because that is where an absent instance turns into a reporting error rather than an inconvenience.

Teams running hybrid or multicloud nodes get a category that did not exist before, including the specific and easily-missed case of a registration conflict caused by cloning a VM that was registered without being deregistered first.

Key takeaways

Six new categories join network connectivity: IAM, agent version, status checks, OS configuration, DHMC and hybrid activation.
Diagnosis is grouped by VPC, which usually reflects where the cause actually lives.
Operating system findings come from console output and cannot be auto-remediated, because fixing them needs the OS access you have lost.
Several categories return guidance rather than a runbook, including NACLs and unattached endpoint subnets.
It runs as Automation runbooks and is billed as such, and requires onboarding the unified Systems Manager console first.