Home› Blog› Week 15 - CloudTrail Organization Trail + Audit Fo…
AWS Weekly Lab AWS Terraform

Nobody could tell me who deleted it

Security Hub and Config can tell you a resource is wrong. Neither can tell you who changed it, when, from where, or what else they touched — and once an auto-remediation has tidied the problem away, the action record is the only evidence it ever happened. This week builds the attribution layer over an organization trail, and gets one of its own alarms badly wrong in a way that is worth more than the rest of the build.

Verified against current vendor documentation on 22 August 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.
AWS Platform Engineering Lab · Week 15 of 52

Lab Overview

During Week 12 of this series, the account's AWS Config recorder was deleted in the middle of a build. Not by me, and not by anything in that week's Terraform — an unrelated project's cleanup script reached across and removed it.

The problem. What I could not do, at any point, was find out who deleted it. Not approximately. Not with effort. The information was gone the moment it happened. Security Hub and GuardDuty find the open security group; Config reports whether a resource is compliant. Both describe state — what is true right now. Attribution is about actions, and auto-remediation destroys that evidence while doing its job.

The objective. An organization-wide CloudTrail with a query layer that can answer "who changed this, when, and from where" after the fact.

The outcome. It answers the question — and the alarm built on top of it fired on every normal login, including mine, for a reason worth the whole week.

DifficultyAdvanced
Lab runtimeNot recorded
CostEstimated — a few cents at lab volume, plus $0.60/month in alarms. See Cost
DeploymentTerraform via HCP Terraform with OIDC
Full lab guideGitHub — prerequisites, Terraform, commands, validation, cleanup

Cloud Services and Concepts

ServicePurpose hereWhat it is
AWS CloudTrail (organization trail)Records management events from every account in the organization into one bucket.One CloudTrail that records API calls for every account in an organization, including accounts created after the trail was made.
Amazon S3Where the trail delivers, under a prefix carrying the organization ID.Object storage: you put files (objects) into buckets and pay for what you store and for each request, with no disk to size in advance.
AWS Glue Data CatalogThe table definition over those logs — defined, not crawled.A catalogue of table definitions — names, columns and types — that query engines read so they know how to interpret files in S3.
Amazon AthenaSeven saved queries a human runs after something has happened, under a workgroup scan ceiling.Runs SQL straight against files in S3 with no database to run. You are billed for the bytes each query scans, so file layout is a cost decision.
AWS Lambda + CloudWatchA daily analyzer turning three "should be zero" questions into alarms.Lambda runs the code; CloudWatch stores the metrics and logs it produces.

State versus action. Config and Security Hub tell you what is wrong. Only an action record tells you who made it wrong — and that distinction is why this week exists after three weeks of state-based tooling.

Partition projection lets Athena compute partitions from the path rather than being told about them. For an organization trail that matters enormously: the alternative is one ALTER TABLE ADD PARTITION per account, per region, per day.

Architecture and Workflow

aws AWS Cloud The organization Organizations management account 6 member accounts no per-account setup One trail covers every account, including ones created tomorrow. 1 CloudTrail org trail · multi-Region The delivered log S3 365-day expiry 2 Athena the query layer 3 Management events only. Data events are off: they bill from the very first copy. The bucket outlives the console's own 90-day history, which is the only reason the trail answers a question asked a year later.
One trail, every account, including the ones that do not exist yet.
  1. An organization trail captures management events from every account.
  2. It delivers to S3 under a prefix including the organization ID, with log file validation on so a log can later be proved unaltered.
  3. A Glue table is defined over that location using five projected partition keys — organization, account, region, and date.
  4. Athena queries it inside a workgroup carrying a 10 GB per-query scan ceiling.
  5. A human runs the seven saved queries after an incident: who changed this, principal timelines, changes made outside Terraform, activity in unexpected regions.
  6. A daily Lambda runs three of those as "should be zero" checks and publishes metrics.
  7. CloudWatch alarms on those metrics notify by SNS.

Design decisions

Five projected keys, because neither AWS recipe fits. The partition-projection example covers a single account in a single region. The organization-wide page handles the extra path segment but only with manual ALTER TABLE, then recommends Lake Formation. So the documented options are a table that cannot see the organization, or one that needs a statement per account per region per day.

A defined table, not a crawler. A crawler costs money to rediscover a layout that is fixed and documented.

Spend as a control. The 10 GB per-query ceiling is enforced at workgroup level, not by convention — and the analyzer's Athena permissions are scoped to that one workgroup, so a broad athena:* cannot bypass the guardrail the design depends on.

Prerequisites

RequirementDetail
AWS OrganizationsA management account, and trusted access for CloudTrail enabled — a manual prerequisite
AWS permissionsCloudTrail, S3, Glue, Athena, Lambda, CloudWatch, SNS
Terraform>= 1.10
Regionus-east-1

Check whether something already took the free copy. AWS gives you one free copy of management events per region, and everybody repeats that. The half that gets left out: if a trail already exists in that region delivering management events, yours is the second copy — and second copies bill at $2.00 per 100,000 events. Look before you create.

Implementation

One trail, one hardened bucket, a defined Glue table, an Athena workgroup with a ceiling, seven saved queries, a daily analyzer and six alarms. The lab guide carries the full Terraform and every query.

week-15-cloudtrail-audit-forensics/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the modules
|   +-- modules/trail/      organization trail, bucket, bucket policy
|   +-- modules/catalog/    Glue table with five projected partition keys
|   +-- modules/analyzer/   workgroup, saved queries, Lambda, alarms
|
+-- queries/                the seven saved queries, as .sql
+-- scripts/                teardown verification
Only the directories that matter. The full repository is on GitHub.
CloudTrail console showing the organization trail: logging, multi-region, applied to the whole organization, log file validation enabled
“Apply trail to my organization: Enabled for all accounts” is the line that matters. Note the log location already carries the AWSLogs/o-…/ segment.

Validation and Evidence

The test is the question that started the week: can it name who did something, after the fact.

TestExpectedActualStatus
Partition projection resolvesRows returned across accounts and regionsReturnedPass
"Who changed this resource?"Principal, time and source IPAll threePass
Changes made outside TerraformIdentifies non-pipeline principalsIdentifiedPass
Activity in unexpected regionsAlarm firesFired and notifiedPass
Console login without MFAFires only on genuinely unprotected loginsFired on six logins, all MFA-protected, all mineFail — see below
Athena results showing PutParameter, AddTagsToResource and DeleteParameter with principal, source IP, user agent and request parameters
Create, tag, delete — with the principal, the source IP, the user agent and the raw request parameters. This is the question that had no answer in Week 12.
Athena results showing mutating API calls made from the CLI rather than Terraform
Changes that did not come from Terraform.
CloudWatch alarm detail showing the unexpected region alarm in ALARM state
The detection firing on something I actually did, not on a fixture.

Security and Cost

Security

Management events only — no object-level access logging, and no per-object cost.

Log file validation on. Signed digests let you later prove a log was not altered after delivery — which is the difference between a record and an audit record.

Bucket policy scoped by aws:SourceArn to this specific trail, against the confused-deputy case. Non-TLS access denied, all public access blocked, BucketOwnerEnforced, SSE-S3, versioned.

Least-privilege IAM. The analyzer's Athena permissions are scoped to the one workgroup carrying the scan ceiling — a broad athena:* would let it bypass the guardrail the design depends on. PutMetricData supports no resource-level permission, so it is constrained by a namespace condition instead.

365-day retention, chosen to outlive the console's own 90-day event history.

Cost

These are rates and an estimate, not a measurement. This post was published before the charges posted and no Cost Explorer figure was recorded afterwards, so it says so rather than presenting arithmetic as a bill.

ItemRateThis build
Management events, first copy to S3freesee the warning in Prerequisites
Additional management event copies$2.00 / 100kthis is the one that applied
Data events$0.10 / 100knot enabled
S3 storage$0.023 / GB-monthpennies at lab volume
Athena$5 / TB scannedcents — every query under 6 MiB
CloudWatch alarms × 6$0.10 / alarm / month$0.60/month

The standing charge is the trail itself, not the queries. Left running it keeps writing every management event from every account in the organization into a bucket, and because the first copy is normally free there is no sharp cost signal to make you notice.

Troubleshooting and Lessons

The MFA alarm fired on every normal login, including mine

Problem. The console-login-without-MFA check went to ALARM reporting six sign-ins with no second factor. Every one was mine, and every one was MFA-protected.
Root cause. IAM Identity Center, SAML and any external identity provider satisfy MFA at the provider and then federate into the console. No second factor happens at the AWS sign-in step — it already happened upstream, where CloudTrail cannot see it. So CloudTrail records MFAUsed = "No" on a login that was fully protected. My query treated a NULL or non-true MFA field as a finding, and the comment above it confidently explained why.
Resolution. Scope the check to IAMUser and Root identity types, where the field means what it appears to mean.
Lesson. A federated sign-in is not an MFA-less sign-in. An alarm that fires on routine activity is worse than no alarm — it teaches people to ignore the channel the real event will arrive in.

CloudWatch alarm email reporting six logins without MFA
Threshold Crossed: 1 datapoint [6.0]. Six federated logins, every one of them behind MFA.

An alarm that latches fails silently

Problem. After the first notification, later detections produced nothing.
Root cause. Maximum over a 24-hour period means one event pins the alarm in ALARM, and every later detection is a non-transition. CloudWatch notifies on transitions.
Resolution. Shorter periods, and treat the metric as a rate rather than a high-water mark.
Lesson. You get one alert, then a dead channel that still looks configured. Worse than a noisy alarm, because nothing indicates it stopped working.

Neither AWS recipe fits an organization trail

Problem. The documented Athena table either cannot see the organization, or needs manual partitions per account per region per day.
Root cause. The published examples target a single account in a single region.
Resolution. Five projected partition keys covering the organization prefix.
Lesson. When both documented options are unusable, the gap is usually one path segment wide.

Cleanup and Key Takeaways

Queue a destroy from HCP, then verify. The organization trail is the one that matters. Left behind it keeps writing every management event from every account into a bucket indefinitely — and because the first copy is normally free, there is no sharp cost signal. Just a bucket that grows, and once the lifecycle rule is destroyed with the rest of the stack, never stops.

The verification script distinguishes three outcomes, not two: gone, still present, and could not check. An expired session that returns an empty list looks exactly like a clean teardown if you only test for emptiness.

Trusted access is deliberately left enabled. It is an organization-level setting, it was a manual prerequisite, and other services may rely on it — removing it is a separate decision from tearing down this week.

Key takeaways

  • State tells you what is wrong; actions tell you who made it wrong. Only an action record answers the second, and auto-remediation destroys that evidence while doing its job.
  • A federated sign-in is not an MFA-less sign-in. Alarming on MFAUsed without scoping to IAMUser and Root means alarming on every normal login.
  • An alarm that fires on routine activity is worse than no alarm. It teaches people to ignore the channel the real event will arrive in.
  • An alarm that latches is worse still, because it fails silently — one alert, then a dead channel that still looks configured.
  • "The first copy is free" has a second half. If a trail already exists in the region, yours is the second copy at $2.00 per 100,000 events.

References

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent