AWS Architecture Series #40 — The controls you cannot add later

Business Challenge

A platform team has an account baseline. It is written down, it is applied by a pipeline, and it is good. During a rush eight months ago an account was created by hand for a proof of concept, the proof of concept became a product, and the account never went through the pipeline. Somebody notices during an audit.

Most of the list applies cleanly. Block Public Access goes on at the account level in one call. Encryption by default is a switch. Region restriction is an SCP the account inherits the moment it is moved into the right OU. A handful of items need remediation work — the unencrypted volumes created over eight months have to be snapshotted and rebuilt — and that is annoying, schedulable, and finite.

Then the team reaches three items and stops. The organisation trail, the Config recorder and GuardDuty can all be switched on this afternoon, and every one of them will start working immediately. None of them will tell you anything about the eight months that already happened. The auditor's question is not "is this account monitored now". It is "show me who changed the security group on this database, and when".

That answer does not exist. It was never written down, and there is no procedure, budget or escalation that creates it now.

Architecture

Sort the baseline into two columns before designing anything. The sort is mechanical: ask what the control produces, and whether producing it late gives you the same artefact.

The reversible column, and what lateness actually costs there

These are settings. Switching them on in month eight produces the same state as switching them on in week one, and the bill for the delay comes in two parts: the exposure during the gap, and remediating whatever was created while the setting was off.

The second part is not always small. You can't directly encrypt existing unencrypted volumes or snapshots. The documented route is to create a snapshot of the volume and then use the snapshot to create a new encrypted volume — per volume, with the downtime that implies, and you cannot change the KMS key that is associated with an existing snapshot or volume either. Eight months of unencrypted volumes is a project. It is, however, a project with an end date.

S3 Block Public Access is the cheapest of them and has a property worth knowing. Applied at the account level it applies to all buckets and access points that are owned by that account, and the settings apply to all AWS Regions globally — one call, no Region loop. And because a bucket policy can be edited by anyone with the permission, AWS makes a specific recommendation about where to apply it: if BlockPublicPolicy is enabled for the entire account rather than for a specific bucket, Amazon S3 blocks public policies even if a user alters the bucket policy to disable this setting.

The path-dependent column, and why it is not about cost

Nothing in this column is hard or expensive to enable. That is what makes it easy to defer, and deferring it is the only mistake in this post that cannot be corrected.

AWS Config is the clearest case. The configuration recorder stores the configuration changes to the resource types in scope as configuration items, and AWS Config creates a configuration item whenever it detects a change to a resource type that it is recording. A configuration history is then a collection of the configuration items for a given resource over any time period — which is to say, it is made of the items that were recorded. Turn the recorder on in month eight and the configuration history for months one to seven is not thin or degraded. It is empty, and it always will be.

GuardDuty is the same shape with a friendlier surface. When you enable it, it automatically starts ingesting the foundational data sources associated with that account — CloudTrail management events, VPC flow logs and DNS logs — and you don't need to enable anything else. Nothing to configure, nothing to wire up, and on first enablement it turns on every protection plan except Runtime Monitoring inside a 30-day free trial. There is no argument against having had it on. There is also no finding it can produce about last March.

CloudTrail: ninety days of grace, and why that is worse than nothing

CloudTrail deserves its own treatment because it is the one that hides the problem. CloudTrail is enabled by default for your AWS account and you automatically have access to the CloudTrail event history, which is a viewable, searchable, downloadable, and immutable record of the past 90 days of management events in an AWS Region, and there are no CloudTrail charges for viewing the event history.

So an account with no trail at all is not obviously broken. Open the console and there is activity, searchable, going back months. Everything looks fine. The documentation is direct about what you actually have: the event history is limited to the past 90 days of events. For an ongoing record of events in your AWS account, create an event data store or a trail.

Three further limits compound it, and each one bites precisely when you are under pressure:

• The event history doesn't provide organization level event aggregation. There is no way to ask a question across the estate.
• An event history search is limited to a single AWS account, only returns events from a single AWS Region, and cannot query multiple attributes. One filter, one Region, one account — and an incident is rarely confined to any of those.
• The Event history page only shows management events. It does not show data events, Insights events, or network activity events. So "who read that object" was never in scope at all.

A rolling ninety-day window that looks like an audit log is the most dangerous shape available. An account with nothing would be noticed. An account with ninety days is noticed on the day someone asks about month four.

Why This Architecture Holds Up

It gives the baseline an argument a checklist cannot make

"Apply the baseline before the account is used" is a policy everyone agrees with and nobody can defend under schedule pressure, because every individual item sounds deferrable. The two-column split changes the conversation: three of these items are not deferrable in the ordinary sense, because deferring them is a decision to never have the data. That is a different sentence from "we will get to it", and it survives contact with a deadline.

It tells you what to automate first

Automation effort is finite and baselines are long. Sorting by column rather than by perceived importance produces an unambiguous order: everything path-dependent goes into account creation itself, because a gap there is permanent, and the reversible items can be enforced by detection and remediation afterwards without loss. That is why these controls belong to account creation and not to a post-creation pipeline that runs "soon".

It explains why organisation-level enablement is the right shape

Every control in the path-dependent column can be applied to the organisation rather than to an account — but not all in the same way, and the difference matters. An organisation trail and GuardDuty with auto-enable genuinely cover accounts that do not exist yet. AWS Config does not work like that: you can have only one customer managed configuration recorder for each AWS account for each AWS Region, so a recorder has to be deployed into every account and Region, and AWS documents Systems Manager Quick Setup for exactly that — to create a customer managed configuration recorder across multiple organizational units (OUs) and AWS Regions using AWS best practices.

An aggregator is not that mechanism and cannot substitute for it. Aggregators provide a read-only view into the source accounts and regions that the aggregator is authorized to view, and do not provide mutating access into a source account or region. It is the console you look at, not the thing that records. Deploy the recorders, then aggregate them — an aggregator over accounts with no recorder is an empty dashboard that looks like a healthy one, which is this post's own failure mode wearing a different hat.

The reversible column stays honest about its own cost

Calling something reversible is not calling it free. The EBS remediation is real work with real downtime, and the S3 masking behaviour means a late Block Public Access does not leave you in the same place as an early one — it leaves you with public policies still written, suppressed by a setting. The model does not claim lateness is cheap. It claims lateness is payable, which the other column is not.

Key Architecture Decisions

Sorting a baseline: the two questions

Reference: the SCP that keeps the recording on

The controls in the first three rows are worth nothing if a member account can switch them off, and the point of an organisation-level control is that the account cannot opt out. This is the deny that makes that true. It sits in the deny sweep, so no identity policy in any member account overrides it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RecordingIsNotOptional",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "config:StopConfigurationRecorder",
        "config:DeleteConfigurationRecorder",
        "config:DeleteDeliveryChannel",
        "guardduty:DeleteDetector",
        "guardduty:DisassociateFromMasterAccount"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn":
            "arn:aws:iam::111122223333:role/OrgSecurityBreakGlass"
        }
      }
    }
  ]
}

Note what this does not do: it grants nobody the ability to manage any of these services. It is a ceiling, so the security tooling account still needs identity policies allowing the actions it legitimately performs — and because SCPs do not apply to the management account, none of this constrains an administrator there, which is one more reason to keep that account empty.

Closing Thought

Most security work is reversible in the sense that matters: you can arrive late, do the work, and end up where you would have been. That is why "we will harden it after launch" is so often a reasonable trade rather than negligence — for most controls, it genuinely is.

A small number of controls are not like that, and they are unusually easy to defer precisely because they are cheap, quiet and produce nothing visible on the day you turn them on. Their entire output is a record, and the record is generated in real time or not at all. There is no supplier, no budget line and no escalation that produces last March.

The test to apply to every item on a baseline: if I switch this on a year late, do I end up with the same thing I would have had? Where the answer is yes, you have a scheduling problem, and scheduling problems have solutions. Where the answer is no, it is not a baseline item at all — it is a property of how accounts are created, and it belongs there rather than on a list someone works through afterwards.

Official AWS Reference

• AWS Documentation — Working with CloudTrail event history
• AWS Documentation — AWS Config terminology and concepts
• AWS Documentation — Working with the configuration recorder
• AWS Documentation — What is Amazon GuardDuty?
• AWS Documentation — Amazon EBS encryption
• AWS Documentation — Blocking public access to your Amazon S3 storage
• AWS Documentation — Service control policies (SCPs)