Home› Blog› Week 3 - Self-Service EC2 Fleet Patching with AWS …
AWS Weekly Lab AWS Terraform

From invisible to fully managed

An inherited fleet with no consistent tagging and no known patch state. A ticket onboards an unmanaged instance or patches the fleet, and no one opens an SSH session.

AWS Platform Engineering Lab · Week 3 of 52

Every enterprise inherits a fleet, not a blank slate. Servers were launched months ago by different teams — no consistent tagging, unknown patch state, no baseline assigned. You can't patch what you can't identify.

Week 3 of my 52-week AWS lab solves both problems: a self-service platform where an ops engineer submits a ServiceNow ticket to either onboard an unmanaged EC2 instance into the managed fleet, or trigger a patch run across the entire environment. The ticket closes itself with a full compliance report. Zero SSH. Zero manual steps.

7SSM Services Used
3Instances Managed
0SSH Connections
100%Patch Compliance
472Patches Installed

Lab Overview

❌ Before
SSH into each instance individually
Run yum update manually
Check output, update spreadsheet
Update ticket manually
Repeat for every instance
No audit trail
✅ After
Submit ServiceNow ticket
SSM patches entire fleet simultaneously
Compliance report auto-generated
Ticket closes automatically with results
Works for 2 or 2000 instances
Full audit trail in Step Functions

Cloud Services and Concepts

ServicePurpose hereWhat it is
AWS Systems ManagerDoes the patching, and reports who is compliantA set of tools for managing instances through an agent already on them: running commands, applying patches and reporting what state each one is in.
Patch Manager and patch baselinesDecides what "patched" means, per groupA baseline is the rule for which patches are approved; a patch group is the tag that says which baseline an instance follows.
Maintenance windowDecides when patching may happenA recurring schedule Systems Manager runs tasks inside, so patching lands in a window someone has agreed to rather than whenever a ticket arrives.
AWS Step FunctionsRoutes the three request types and keeps the audit trailRuns a workflow as an explicit state machine, so each step, its retry behaviour and its failure path are declared rather than buried in code.
AWS LambdaReceives the webhook, onboards, reports backRuns a piece of code on demand with no server to manage. You are billed per request and per millisecond it runs.
Amazon API GatewayThe signed endpoint ServiceNow callsA managed front door that accepts HTTP requests and forwards them to a backend.
VPC interface endpointsLets private instances reach Systems Manager with no NAT gatewayPrivate doors from your network to an AWS service. Traffic reaches the service without crossing the internet, and no per-GB NAT charge applies.

One idea runs through the build: in Systems Manager a tag is not documentation, it is configuration. PatchGroup is what binds an instance to a baseline, so an untagged instance is not mis-patched — it is unmanaged, and reports nothing at all.

Architecture and Workflow

aws AWS Cloud ServiceNow catalog request 1 Region: us-east-1 API Gateway signed POST 2 webhook receiver validates 3 Step Functions 4 states 4 SSM Automation onboard or patch EC2 fleet no SSH No SSH anywhere: the SSM agent is the only way in, so access is an IAM decision rather than a key on someone's laptop. The ticket closes with the compliance report attached. 5

Prerequisites

RequirementWhyHow to check
The SSM agent running, with an instance profile that allows itSystems Manager reaches an instance through its agent and the agent's role. Without the role the instance is simply invisible, with no error anywhereIt appears in Fleet Manager as Online
A network path to the Systems Manager endpointsThe agent has to reach the service. This build uses VPC interface endpoints rather than a NAT gateway, which removes the per-GB charge entirelyThe instance registers without a public route
A PatchGroup tag on anything you expect to be patchedThe tag is what associates an instance with a baseline. An untagged instance reports no compliance state rather than a bad onePatch Manager lists the instance against a baseline
A shared secret for the webhookThe endpoint accepts only signed requestsHeld as a sensitive variable, never in the repo

Implementation

The Onboard Story — Making an Invisible Instance Visible

This is the most important feature of Week 3. I created an EC2 instance manually — the way most instances in a real enterprise already exist. No PatchGroup tag, no ManagedBy tag. Completely invisible to Patch Manager from a management perspective.

EC2 Tags tab showing only Name tag — no PatchGroup, no ManagedBy
EC2 Tags tab showing only Name tag — no PatchGroup, no ManagedBy

In Patch Manager, the instance appears but has no patch baseline assigned — Patch configuration name shows "-". It's visible but unmanaged.

One onboard ticket changes everything. The SSM Automation runbook runs 5 steps:

  1. Wait for SSM agent to come online
  2. Apply tags — PatchGroup=fleet-mgmt-dev-linux, ManagedBy=fleet-mgmt-dev, OnboardedAt=timestamp
  3. Run baseline config via Run Command (yum update, confirm SSM agent running)
  4. Run initial patch scan to establish compliance baseline
  5. Collect compliance summary and return results

After onboard — all three tags applied automatically. The instance is now a first-class member of the managed fleet.

EC2 Tags tab after onboard — PatchGroup, ManagedBy, and OnboardedAt all applied
EC2 Tags tab after onboard — PatchGroup, ManagedBy, and OnboardedAt all applied
Patch Manager showing 3 instances — manual-test-instance now alongside fleet instances
Patch Manager showing 3 instances — manual-test-instance now alongside fleet instances

Patching All Three Instances in One Operation

With the manual instance onboarded, all three share PatchGroup=fleet-mgmt-dev-linux. A single Install request patches all of them simultaneously — the platform can't tell which instances were ASG-launched and which was manually created.

Result: all 3 instances fully patched. 472 patches installed across the fleet. 0 missing. 0 failed.

Patch Manager Compliance reporting — all 3 instances Compliant, MissingCount 0
Patch Manager Compliance reporting — all 3 instances Compliant, MissingCount 0
💡 The result: 3 instances — 2 from the ASG, 1 created manually — all patched to 100% compliance in a single Step Functions execution. Zero SSH connections made.

ServiceNow Integration — End to End

When fully wired, a user never touches AWS. They open the Service Catalog, fill in a form, and submit. The entire flow is invisible to them.

The status_updater Lambda calls back to ServiceNow and closes the ticket with a full compliance report — instance count, compliant/non-compliant, patch details, link to Patch Manager console.

ServiceNow RITM closed with compliance report in work notes — 6 compliant, 472 patches installed, 100% rate
ServiceNow RITM closed with compliance report in work notes — 6 compliant, 472 patches installed, 100% rate

No NAT Gateway — VPC Endpoints Only

This is the enterprise pattern. NAT gateways cost ~$32/month and route all SSM traffic through the public internet. SSM is an AWS service — there's no reason for that traffic to leave the AWS network.

EndpointPurposeType
ssmMain SSM API — commands, parameters, registrationsInterface
ssmmessagesSession Manager WebSocket channelInterface
ec2messagesRun Command delivery channelInterface
logsCloudWatch Logs for Lambda and SSM agentInterface
s3Patch downloads and log uploadsGateway
⚠️ Gotcha: Not all AZs in us-east-1 support VPC interface endpoints. Use an aws_vpc_endpoint_service data source to filter to only supported AZs before creating subnets — otherwise Terraform fails with a cryptic AZ error.

Validation and Evidence

Each row traces to a figure here. The lab was destroyed in June 2026; these are the results recorded then.

TestExpectedActualStatus
Starting state is honestInstances that have never been scanned report as suchPatch Manager showed Never reported before any scan ranPass
An unmanaged instance is invisible, not wrongAn instance with only a Name tag has no baselineNo PatchGroup, no ManagedBy, and no baseline assigned in Patch ManagerPass
Onboarding tags itThe workflow applies the tags that make it managedPatchGroup, ManagedBy and OnboardedAt all present afterwardsPass
The fleet grows by onePatch Manager counts the new instance with the othersThree instances listed, the manual one alongside the fleetPass
Patching reaches complianceEvery instance compliant, nothing missingAll three Compliant, MissingCount zeroPass
Every request is auditableOne execution per request, each with a timestampAll three executions SUCCEEDED in the executions listPass
The requester gets the answerThe ticket closes carrying the compliance resultClosed with the report in its work notes: 6 compliant, 472 patches installedPass

The Demo — Watch It Work

Step 1 — Deploy: 80+ Resources, One Command

Terraform deploys the entire platform: VPC with SSM interface endpoints (no NAT gateway), EC2 fleet in private subnets, 4 Lambda functions, Step Functions state machine, patch baselines for Amazon Linux 2023 and Windows Server, maintenance window, and API Gateway.

Step 2 — Fleet Manager: Instances Online

Within 5 minutes of launch, both ASG instances register with SSM automatically — no configuration needed. Fleet Manager gives a real-time view: online status, OS type, SSM agent version, last ping time. No SSH required to confirm they're healthy.

Step 3 — Patch Manager: The Before State

Fresh instances — no scan has run yet. This is the starting point the platform is designed to fix.

Patch Manager Compliance reporting — instances showing Never reported before any scan
Patch Manager Compliance reporting — instances showing Never reported before any scan

Step 4 — Simulate ServiceNow: Trigger a Patch Scan

A single curl command simulates what a ServiceNow business rule does — signs the payload with HMAC, posts to API Gateway, and starts a Step Functions execution.

BODY='{"ticket_id":"RITM0030001","request_type":"patch",
      "patch_group":"fleet-mgmt-dev-linux","operation":"Scan",
      "team":"platform-engineering","requested_by":"jay.katta"}'
SIG="sha256=$(echo -n "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')"

curl -X POST "$API_URL" \
  -H "Content-Type: application/json" \
  -H "x-servicenow-signature: $SIG" \
  -d "$BODY"

Full Audit Trail + CloudWatch Dashboard

Every operation is traceable. Step Functions records every execution with the ticket ID, start time, and result. In production this maps directly to a ServiceNow RITM — you can look up any patch run and know exactly who requested it, when it ran, and what the outcome was.

Step Functions executions list showing all 3 executions SUCCEEDED with timestamps
Step Functions executions list showing all 3 executions SUCCEEDED with timestamps

Security and Cost

ResourceCost/Month (Running)Cost (Destroyed)
EC2 t3.micro x2~$17$0
VPC endpoints x4 (interface)~$29$0
CloudWatch~$2$0
SSM (core features)$0$0
Lambda (minimal traffic)~$0$0
S3 session logs~$0.02$0
Total~$48/month$0
💡 VPC endpoints are the biggest cost driver. For a lab you can swap them for a NAT Gateway, but VPC endpoints are the enterprise pattern — SSM traffic never leaves the AWS network. Run sh scripts/cleanup.sh when done for the day.

Troubleshooting and Lessons

Problem. Every SSM Parameter Store path the project tried to create was rejected.
Root cause. AWS reserves any parameter path beginning /ssm, case-insensitively, and the project was named ssm-fleet — so every path began /ssm-fleet/.
Resolution. Renamed the project to fleet-mgmt.
Lesson. A project name becomes a path prefix, and a reserved prefix is a naming decision you cannot undo cheaply later.

Problem. Terraform failed with a cryptic error while creating VPC endpoints.
Root cause. Not every Availability Zone offers the Systems Manager interface endpoints — us-east-1e does not — and slice(AZs, 0, 2) was picking whichever two came first.
Resolution. Query aws_vpc_endpoint_service and build subnets only in the zones that actually offer the endpoint.
Lesson. "Pick the first two AZs" assumes the zones are interchangeable. For endpoints they are not.

Problem. The patch-fleet automation document failed validation.
Root cause. CommandId is a reserved output name in SSM Automation, and the document declared a custom output using it.
Resolution. Renamed it to PatchCommandId.
Lesson. Reserved names are rarely listed where you are looking; the validation error is the documentation.

Problem. Instances would not launch at all.
Root cause. The AL2023 AMI snapshot needs at least 30 GB and the launch template asked for 20.
Resolution. 30 GB, on gp2 — gp3 is not offered in every us-east-1 zone either.
Lesson. The second AZ-availability surprise in one build. Zone capability is worth checking before the template, not after it fails.

Problem. InvalidAutomationExecutionParametersException on every patch request from ServiceNow.
Root cause. ServiceNow sent scan; SSM Automation accepts Scan or Install, capitalised.
Resolution. Normalise the operation in the orchestrator rather than relying on the sender's casing.
Lesson. Anything arriving from another system is input to be validated and normalised, not a value to pass through.

Cleanup and Key Takeaways

Key Takeaways

SSM is a platform, not a tool. Seven distinct services — Fleet Manager, Patch Manager, Run Command, Session Manager, State Manager, Automation, Inventory — each solving a different operations problem. Most teams use one or two. Using all seven turns your EC2 fleet into a properly managed, audited, self-healing environment.

Tags are the source of truth for Patch Manager. No PatchGroup tag = no baseline = invisible to Patch Manager. The onboard workflow exists specifically to fix this for instances that predate your tagging strategy.

Session Manager eliminates SSH entirely. No port 22, no key pairs, no bastion hosts. Every session is logged to S3 and CloudWatch. For compliance-heavy environments this is not a nice-to-have — it's the only acceptable access pattern.

14 years as a DBA showed me what manual patching costs. An ops engineer patching 200 servers manually takes a full day — and produces no audit trail. This platform patches 200 servers in the same time it takes to patch 2, and every operation is traceable to a ticket ID and timestamp.


What's New in SSM Patch Manager (2025–2026)

Since this post was written, two updates are worth knowing:

  • Patch now (on-demand) — you can now trigger a scan or install operation immediately on any managed node without setting up a maintenance window first. Useful for emergency patching outside the scheduled window.
  • Quick Setup patch policies — AWS now recommends using Quick Setup to configure patching across your organization. It eliminates manual patch group setup and provides centralized control from a single place.

What's Next — Week 4

Week 3 automates patching. Week 4 answers the harder question: did it work across the entire estate?

Week 4 is a Fleet Intelligence Platform — using AWS Glue to crawl SSM inventory and patch data synced to S3, transform it with PySpark ETL, and make it queryable via Athena SQL. Instead of "2 instances compliant" you get: "Show me every instance that hasn't been patched in 30 days, sorted by severity."


All code available on GitHub: katta698/AWS-Platform-Engineering-Lab. Questions? Drop a comment below.

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent