📋 In This Post
Why — The Problem This Solves
The dangerous cloud misconfigurations are also the most common and the most mechanical: a security group left open to 0.0.0.0/0 on SSH or RDP, an S3 bucket with public access enabled, encryption switched off. Tools like GuardDuty and Security Hub are very good at finding these. The gap is what happens next — the finding lands in a console, and then it sits there.
Mean-time-to-remediate for these boring-but-critical issues is measured in hours or days, which is exactly the window an attacker needs. And teams don't already auto-remediate because it's frightening: one bad rule can revoke a legitimate security group or break production, so the finding gets a "we'll review it" that never comes.
This week closes the loop. For a defined set of clear-cut findings the fix happens automatically in seconds and the finding is marked resolved — while the judgement calls still reach a human. The scary part (blast radius) is handled by a single, boring guardrail: nothing is ever mutated unless it carries an explicit opt-in tag.
What You Need to Know — Skills & Tools
The one decision that shaped everything: classic CSPM, not the new unified Security Hub
At re:Invent in December 2025 AWS released a re-launched, unified "AWS Security Hub" (v2 APIs) that correlates signals from GuardDuty, Inspector, Macie and the classic Security Hub into one detection-and-response view. It's a genuinely nice evolution — but as of mid-2026 the Terraform AWS provider does not yet stably support it (the aws_securityhub_account_v2 resource is still an open feature request).
That's fine, because the thing we're building — findings → automation rules → EventBridge → Lambda → writeback — is entirely Security Hub CSPM (the classic service, now named that), which the provider fully supports. v2 is a correlation and analytics layer sitting above the finding stream; it doesn't change how you react to findings. So this build uses classic CSPM in Terraform, and you can enable v2 alongside it in the console for the richer dashboards.
The Security Hub controls this stack remediates (EC2.13/14/19, S3.2/3/8) are backed by AWS Config. Without a configuration recorder actively recording security groups and S3 buckets, those controls report NO_DATA, no findings are generated, and nothing is remediated — the pipeline deploys cleanly but does nothing. This stack does not provision Config (an account typically has one centrally-managed recorder; adding a second conflicts). Confirm one is recording before you deploy: aws configservice describe-configuration-recorder-status should return recording=True, lastStatus=SUCCESS.
Why Lambda — and not Step Functions or ECS
Each remediation is one to a few API calls, runs in seconds, is stateless, event-triggered, and has near-zero concurrency: revoke a security-group rule, or set Block Public Access on a bucket. That is precisely the shape Lambda exists for. Step Functions would earn its place only if a remediation needed multi-step orchestration, waits, or a human-approval timeout; ECS/Fargate only if a remediation were long-running or had heavy dependencies. Neither applies here, so bringing either in would be weight without benefit.
Two more choices worth stating plainly. EventBridge, not a direct Security Hub → Lambda call, because Security Hub has no native Lambda target — EventBridge is the integration point, and it also gives content-based filtering (each Lambda only ever receives the finding class it fixes) and parallel fan-out to SNS. And both GuardDuty and Security Hub, not one, because they answer different questions: CSPM asks "is this bucket configured to be public?" while GuardDuty asks "is someone exfiltrating from it right now?" GuardDuty's findings flow into Security Hub too, so it stays one stream to react to.
Architecture — How It Fits Together
GuardDuty and Security Hub CSPM produce findings; a Security Hub automation rule enriches them before anything is routed; EventBridge matches specific finding types and fans them out; three Lambdas do the work — two fix, one notifies; and every automated action is written back to the finding so Security Hub reflects reality. Failures land in a dead-letter queue that alarms.
Detect
GuardDuty analyses CloudTrail, VPC Flow Logs and DNS for active threats (Extended Threat Detection is auto-enabled at no cost). Security Hub CSPM runs the AWS Foundational Security Best Practices standard and ingests GuardDuty's findings — all normalized to one schema, the AWS Security Finding Format (ASFF).
Enrich (before routing)
A Security Hub automation rule runs inside Security Hub, before any EventBridge rule fires. It escalates any failed control on a resource tagged environment=production to CRITICAL — so downstream paging is smarter without touching the Lambdas.
Route
EventBridge rules match on detail.findings.Compliance.SecurityControlId (EC2.13/14/19 → the SG remediator, S3.2/3/8 → the S3 remediator) and on the native aws.guardduty event (→ the threat notifier). Crucially, they match only Workflow.Status = NEW — more on why that word matters in Challenges.
Act — but only on opt-in
The SG remediator revokes just the world-open ingress on high-risk ports; the S3 remediator applies all four Block Public Access settings. Both act only if the resource carries auto-remediate=true — enforced in code and in IAM. Anything untagged is left untouched and a human is notified instead. The threat notifier never mutates: active threats are judgement calls.
Write back & be safe
Every action is written back via BatchUpdateFindings (workflow status → RESOLVED or NOTIFIED) so the console reflects reality, and results go to SNS. Any failed asynchronous invocation lands in a 14-day SQS dead-letter queue that raises a CloudWatch alarm — a dropped security action is worse than a slow one.
How We Built It — Step by Step
The stack is 30 Terraform resources across three modules (securityhub, guardduty, remediation), deployed through HCP Terraform. The walkthrough below follows the order things actually come up and get tested.
Step 1 — Enable Security Hub CSPM and the FSBP standard
Enable the account, subscribe the AWS Foundational Security Best Practices standard (the source of the config findings we remediate), add a cross-region finding aggregator, and set control_finding_generator = "SECURITY_CONTROL" so findings carry a populated Compliance.SecurityControlId — the exact field the EventBridge rules key on.
Step 2 — Enable GuardDuty
A single detector turns on foundational threat detection (CloudTrail, VPC Flow Logs, DNS) with no extra data-source cost; Extended Threat Detection comes along for free. Findings publish to EventBridge every 15 minutes.
Step 3 — Deploy the remediation backbone
The remediation module uses a for_each over a map of three Lambda definitions, so each function gets its own least-privilege role, log group, EventBridge rule and invoke permission from one block — plus a shared SNS topic, a shared SQS dead-letter queue, and a CloudWatch alarm on DLQ depth.
Step 4 — Watch the security-group remediation, end to end
To prove the loop, a small script deliberately creates an offending security group — SSH (22) and RDP (3389) open to 0.0.0.0/0 — tagged auto-remediate=true.
Security Hub raises the FSBP finding; EventBridge routes it to sg_remediator, which confirms the tag, revokes exactly the world-open rules on the high-risk ports, and writes the finding back to RESOLVED. Start to finish, this took about 90 seconds.
Step 5 — And the S3 remediation
Same shape for a deliberately public bucket, also tagged for opt-in: the finding fires, s3_remediator applies all four Block Public Access settings, and marks the finding RESOLVED.
Step 6 — The threat path: notify, don't touch
GuardDuty sample findings (a credential-exfiltration and a crypto-mining finding) exercise the third Lambda. threat_notifier formats the finding and publishes it to SNS for a human — and deliberately does not remediate. Killing the wrong instance or key mid-incident can be worse than the threat itself.
Step 7 — The guardrail, proven
This is the most important screenshot in the post. When Security Hub flagged a real, pre-existing public bucket in the account that was not tagged for opt-in, the remediator did exactly what it should: nothing. It sent a review notification and left the bucket alone.
Challenges — What Actually Went Wrong
Three problems were worth the scar tissue. The last one only surfaced because the pipeline was tested against the account's real findings, not just synthetic ones — and it's the most important lesson of the week.
The very first apply errored on aws_securityhub_standards_subscription: "timeout while waiting for state to become READY, INCOMPLETE (last state: PENDING, timeout: 3m0s)". The FSBP standard has roughly 300 controls, and on a cold account it sits in PENDING for longer than the provider's 3-minute default before it finishes initializing. Worse, each failed apply disabled and re-enabled the standard, resetting the clock — a loop.
Raise the create timeout on the resource: timeouts { create = "15m" }. Subscribing a Security Hub standard from a fresh account is not instantaneous — budget for it, or the first apply looks broken when it isn't.
After pushing the timeout fix, I queued a run through the HCP API — and it timed out at 3 minutes again, with the 15-minute setting nowhere in sight. The cause: a plain run created via the HCP API reuses the workspace's latest already-ingressed configuration version; it does not fetch the new commit. Both runs had pinned to the pre-fix code.
FixBind the run explicitly to the new commit's configuration version (or let the VCS webhook ingress it first and confirm the run's version matches the commit you intend). With HCP VCS workspaces, "queue a run via the API" is not the same as "deploy my latest commit" — verify which one you're actually applying.
Live testing surfaced the real one. The S3 remediator started firing about eight times a minute, indefinitely, against a pre-existing untagged bucket — each invocation sending a notification email. The EventBridge rule matched Workflow.Status of both NEW and NOTIFIED. So on the notify path the Lambda called BatchUpdateFindings to set the status to NOTIFIED — and that update was re-emitted as a "Findings - Imported" event, which re-matched the rule, which re-invoked the Lambda, which set NOTIFIED again. A handler was triggering itself off its own writeback.
Match Workflow.Status = ["NEW"] only. A handler that writes back to the same finding store it is triggered by must keep its writeback status outside its own trigger pattern — trigger on NEW, and make RESOLVED/NOTIFIED fall outside the match. Security Hub resets a recurring finding back to NEW, so genuine recurrences are still caught. This is the single most important design rule for any find-and-act automation, and it's the kind of bug that only a real end-to-end run reveals.
Security — Guardrails on the Automation
The whole point of auto-remediation is that it acts without a human in the loop — so the safety has to be structural, not procedural.
Tag-scoped opt-in, enforced twice
Mutating remediations only touch resources tagged auto-remediate=true. The SG remediator enforces this in two independent places: the code checks the tag, and its IAM policy adds an aws:ResourceTag condition so the RevokeSecurityGroupIngress API itself refuses to act on anything untagged.
Least privilege per function
Each Lambda has its own role scoped to exactly the actions it needs. The threat notifier has no mutating permissions at all — it can publish to SNS and nothing else.
Surgical, idempotent changes
The SG remediator strips only the 0.0.0.0/0 and ::/0 ingress on high-risk ports, leaving legitimate scoped rules untouched, and treats an already-removed rule as success. Re-running it is a no-op.
No silent failures
Every failed asynchronous invocation goes to a 14-day dead-letter queue that alarms to SNS. Threat findings are never auto-mutated. And every automated action is written back to the finding, so there's always an audit trail of what changed and why.
Cost
For a single, low-traffic account in us-east-1. Prices as of July 2026 — verify at the Security Hub and GuardDuty pricing pages.
| Item | Estimate |
|---|---|
| GuardDuty foundational (CloudTrail $4/M events, VPC/DNS ~$1/GB) | ~$2–5 / month |
| Security Hub CSPM checks ($0.0010/check; first 10k findings/mo free) | ~$3–15 / month |
| EventBridge + Lambda + SNS + SQS | pennies (well within free tier) |
| 30-day free trial (both Security Hub and GuardDuty) | covers the lab window |
| Destroyed | $0 |
Cleanup
Queue a destroy plan on the week-11-dev workspace in HCP — it disables Security Hub CSPM and the GuardDuty detector and removes the Lambdas, rules, SNS topic and DLQ. Destroying is inexpensive to reverse: one apply rebuilds the whole stack in about ten minutes. Any deliberately-insecure test resources are torn down with a small cleanup script.
References
Key Takeaways
- The loop-break is the whole game. A find-and-act automation whose action updates the thing that triggered it will feed itself forever unless the writeback status falls outside the trigger pattern. Trigger on
NEWonly. - Make auto-remediation opt-in, not opt-out. Tag-scoping (enforced in both code and IAM) is what turns "terrifying" into "safe to run in production" — an untagged resource is reported, never touched.
- Fix the mechanical, escalate the judgement calls. Config misconfigurations get auto-closed; active threats get a human. Same backbone, different verbs.
- Know your prerequisites. Security Hub controls need AWS Config recording, and standards take minutes to initialize. Neither is obvious until the pipeline silently does nothing.
- Real end-to-end tests find what unit tests can't. The feedback loop was invisible to synthetic tests — it only appeared when the pipeline met the account's actual findings.
Comments