📋 In This Post
Lab Overview
During Week 12 of this series, the account's AWS Config recorder was deleted in the middle of a build. Not by me, and not by anything in that week's Terraform — an unrelated project's cleanup script reached across and removed it.
The problem. What I could not do, at any point, was find out who deleted it. Not approximately. Not with effort. The information was gone the moment it happened. Security Hub and GuardDuty find the open security group; Config reports whether a resource is compliant. Both describe state — what is true right now. Attribution is about actions, and auto-remediation destroys that evidence while doing its job.
The objective. An organization-wide CloudTrail with a query layer that can answer "who changed this, when, and from where" after the fact.
The outcome. It answers the question — and the alarm built on top of it fired on every normal login, including mine, for a reason worth the whole week.
| Difficulty | Advanced |
| Lab runtime | Not recorded |
| Cost | Estimated — a few cents at lab volume, plus $0.60/month in alarms. See Cost |
| Deployment | Terraform via HCP Terraform with OIDC |
| Full lab guide | GitHub — prerequisites, Terraform, commands, validation, cleanup |
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| AWS CloudTrail (organization trail) | Records management events from every account in the organization into one bucket. | One CloudTrail that records API calls for every account in an organization, including accounts created after the trail was made. |
| Amazon S3 | Where the trail delivers, under a prefix carrying the organization ID. | Object storage: you put files (objects) into buckets and pay for what you store and for each request, with no disk to size in advance. |
| AWS Glue Data Catalog | The table definition over those logs — defined, not crawled. | A catalogue of table definitions — names, columns and types — that query engines read so they know how to interpret files in S3. |
| Amazon Athena | Seven saved queries a human runs after something has happened, under a workgroup scan ceiling. | Runs SQL straight against files in S3 with no database to run. You are billed for the bytes each query scans, so file layout is a cost decision. |
| AWS Lambda + CloudWatch | A daily analyzer turning three "should be zero" questions into alarms. | Lambda runs the code; CloudWatch stores the metrics and logs it produces. |
State versus action. Config and Security Hub tell you what is wrong. Only an action record tells you who made it wrong — and that distinction is why this week exists after three weeks of state-based tooling.
Partition projection lets Athena compute partitions from the path rather than
being told about them. For an organization trail that matters enormously: the alternative is one
ALTER TABLE ADD PARTITION per account, per region, per day.
Architecture and Workflow
- An organization trail captures management events from every account.
- It delivers to S3 under a prefix including the organization ID, with log file validation on so a log can later be proved unaltered.
- A Glue table is defined over that location using five projected partition keys — organization, account, region, and date.
- Athena queries it inside a workgroup carrying a 10 GB per-query scan ceiling.
- A human runs the seven saved queries after an incident: who changed this, principal timelines, changes made outside Terraform, activity in unexpected regions.
- A daily Lambda runs three of those as "should be zero" checks and publishes metrics.
- CloudWatch alarms on those metrics notify by SNS.
Design decisions
Five projected keys, because neither AWS recipe fits. The
partition-projection example covers a single account in a single region. The organization-wide
page handles the extra path segment but only with manual ALTER TABLE, then
recommends Lake Formation. So the documented options are a table that cannot see the
organization, or one that needs a statement per account per region per day.
A defined table, not a crawler. A crawler costs money to rediscover a layout that is fixed and documented.
Spend as a control. The 10 GB per-query ceiling is enforced at workgroup
level, not by convention — and the analyzer's Athena permissions are scoped to that one
workgroup, so a broad athena:* cannot bypass the guardrail the design depends
on.
Prerequisites
| Requirement | Detail |
|---|---|
| AWS Organizations | A management account, and trusted access for CloudTrail enabled — a manual prerequisite |
| AWS permissions | CloudTrail, S3, Glue, Athena, Lambda, CloudWatch, SNS |
| Terraform | >= 1.10 |
| Region | us-east-1 |
Check whether something already took the free copy. AWS gives you one free copy of management events per region, and everybody repeats that. The half that gets left out: if a trail already exists in that region delivering management events, yours is the second copy — and second copies bill at $2.00 per 100,000 events. Look before you create.
Implementation
One trail, one hardened bucket, a defined Glue table, an Athena workgroup with a ceiling, seven saved queries, a daily analyzer and six alarms. The lab guide carries the full Terraform and every query.
week-15-cloudtrail-audit-forensics/
|
+-- terraform/
| +-- environments/dev/ HCP backend, OIDC, wires the modules
| +-- modules/trail/ organization trail, bucket, bucket policy
| +-- modules/catalog/ Glue table with five projected partition keys
| +-- modules/analyzer/ workgroup, saved queries, Lambda, alarms
|
+-- queries/ the seven saved queries, as .sql
+-- scripts/ teardown verification
AWSLogs/o-…/ segment.Validation and Evidence
The test is the question that started the week: can it name who did something, after the fact.
| Test | Expected | Actual | Status |
|---|---|---|---|
| Partition projection resolves | Rows returned across accounts and regions | Returned | Pass |
| "Who changed this resource?" | Principal, time and source IP | All three | Pass |
| Changes made outside Terraform | Identifies non-pipeline principals | Identified | Pass |
| Activity in unexpected regions | Alarm fires | Fired and notified | Pass |
| Console login without MFA | Fires only on genuinely unprotected logins | Fired on six logins, all MFA-protected, all mine | Fail — see below |
Security and Cost
Security
Management events only — no object-level access logging, and no per-object cost.
Log file validation on. Signed digests let you later prove a log was not altered after delivery — which is the difference between a record and an audit record.
Bucket policy scoped by aws:SourceArn to this specific trail,
against the confused-deputy case. Non-TLS access denied, all public access blocked,
BucketOwnerEnforced, SSE-S3, versioned.
Least-privilege IAM. The analyzer's Athena permissions are scoped to the one
workgroup carrying the scan ceiling — a broad athena:* would let it bypass the
guardrail the design depends on. PutMetricData supports no resource-level
permission, so it is constrained by a namespace condition instead.
365-day retention, chosen to outlive the console's own 90-day event history.
Cost
These are rates and an estimate, not a measurement. This post was published before the charges posted and no Cost Explorer figure was recorded afterwards, so it says so rather than presenting arithmetic as a bill.
| Item | Rate | This build |
|---|---|---|
| Management events, first copy to S3 | free | see the warning in Prerequisites |
| Additional management event copies | $2.00 / 100k | this is the one that applied |
| Data events | $0.10 / 100k | not enabled |
| S3 storage | $0.023 / GB-month | pennies at lab volume |
| Athena | $5 / TB scanned | cents — every query under 6 MiB |
| CloudWatch alarms × 6 | $0.10 / alarm / month | $0.60/month |
The standing charge is the trail itself, not the queries. Left running it keeps writing every management event from every account in the organization into a bucket, and because the first copy is normally free there is no sharp cost signal to make you notice.
Troubleshooting and Lessons
The MFA alarm fired on every normal login, including mine
Problem. The console-login-without-MFA check went to ALARM
reporting six sign-ins with no second factor. Every one was mine, and every one
was MFA-protected.
Root cause. IAM Identity Center, SAML and any external identity provider
satisfy MFA at the provider and then federate into the console. No second factor happens
at the AWS sign-in step — it already happened upstream, where CloudTrail cannot see it. So
CloudTrail records MFAUsed = "No" on a login that was fully
protected. My query treated a NULL or non-true MFA field as a finding, and
the comment above it confidently explained why.
Resolution. Scope the check to IAMUser and Root
identity types, where the field means what it appears to mean.
Lesson. A federated sign-in is not an MFA-less sign-in. An
alarm that fires on routine activity is worse than no alarm — it teaches people to ignore
the channel the real event will arrive in.
Threshold Crossed: 1 datapoint [6.0]. Six federated logins, every one of them behind MFA.An alarm that latches fails silently
Problem. After the first notification, later detections produced
nothing.
Root cause. Maximum over a 24-hour period means one event pins the
alarm in ALARM, and every later detection is a non-transition. CloudWatch notifies
on transitions.
Resolution. Shorter periods, and treat the metric as a rate rather than a
high-water mark.
Lesson. You get one alert, then a dead channel that still looks
configured. Worse than a noisy alarm, because nothing indicates it stopped
working.
Neither AWS recipe fits an organization trail
Problem. The documented Athena table either cannot see the organization, or
needs manual partitions per account per region per day.
Root cause. The published examples target a single account in a single
region.
Resolution. Five projected partition keys covering the organization prefix.
Lesson. When both documented options are unusable, the gap is usually one
path segment wide.
Cleanup and Key Takeaways
Queue a destroy from HCP, then verify. The organization trail is the one that matters. Left behind it keeps writing every management event from every account into a bucket indefinitely — and because the first copy is normally free, there is no sharp cost signal. Just a bucket that grows, and once the lifecycle rule is destroyed with the rest of the stack, never stops.
The verification script distinguishes three outcomes, not two: gone, still present, and could not check. An expired session that returns an empty list looks exactly like a clean teardown if you only test for emptiness.
Trusted access is deliberately left enabled. It is an organization-level setting, it was a manual prerequisite, and other services may rely on it — removing it is a separate decision from tearing down this week.
Key takeaways
- State tells you what is wrong; actions tell you who made it wrong. Only an action record answers the second, and auto-remediation destroys that evidence while doing its job.
- A federated sign-in is not an MFA-less sign-in. Alarming on
MFAUsedwithout scoping toIAMUserandRootmeans alarming on every normal login. - An alarm that fires on routine activity is worse than no alarm. It teaches people to ignore the channel the real event will arrive in.
- An alarm that latches is worse still, because it fails silently — one alert, then a dead channel that still looks configured.
- "The first copy is free" has a second half. If a trail already exists in the region, yours is the second copy at $2.00 per 100,000 events.
Comments