Homeβ€Ί Blogβ€Ί AWS Architecture Series #116 β€” Six policies, one API, and only one of them deletes data…
AWS Architecture AWS Architecture Series

AWS Architecture Series #116 β€” Six policies, one API, and only one of them deletes data

PutAccountPolicy creates six different policy types with one call, and the rule for what happens when an account-level policy meets a log-group-level policy is different for every one of them. Data protection is cumulative, subscription filters are additive, transformers and field indexes defer to the log group, metric extraction has no log-group equivalent - and retention inverts all of it: an account-level retention policy replaces the log group's own setting even when the log group's period is longer, which deletes log events older than the new period. The selection criteria are different per type too, so the exclusion you need is expressible for some policy types and not for others.

Verified against current vendor documentation on 17 November 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

The last three posts each ended at the same door. #113's field indexes have an account-level form. #114's log classes are set per log group but governed in bulk. #115's subscription filters are account-level or log-group-level, and the exclusions that prevent a recursion loop live in selectionCriteria. All of it runs through one API:

“Creates an account-level data protection policy, subscription filter policy, field index policy, transformer policy, metric extraction policy, or retention policy that applies to all log groups, a subset of log groups, or a data source name and type combination in the account.”

Six nouns in one sentence, behind one operation, with one policyType parameter. The shape invites you to learn it once. The documentation then spends several screens explaining that almost nothing generalises between them.

1Five policy types defer to the log group. Retention overrides it, and that means deletion

Every other type resolves a conflict in the log group's favour, or in favour of doing more rather than less: for data protection, “The two policies are cumulative. Any sensitive term specified in either the account-level or the log-group-level policy is masked”; “Account-level and log-group-level subscription filters are additive”; “A log-group-level transformer overrides the account-level transformer”; and for indexes, a log-group policy causes prefix-scoped account policies to be “ignored for that log group”.

Then retention:

“An account-level retention policy replaces the retention period set on a log group, including when the log group's own retention period is longer.”

With AWS supplying the worked example, in bold type on its own page: “For example, if a log group is set to retain log events for 10 years and you create a retention policy that sets a 90-day retention period for that log group, CloudWatch Logs deletes the log events in that log group that are older than 90 days.”

Nothing else in this API removes data. A transformer reshapes new events, a subscription copies them, an index speeds up queries, a data protection policy masks on ingestion. Retention is the one that reaches backwards into what you already have, and it does so by overriding the explicit choice a log group's owner made. A team that set ten years because a regulator asked for ten years has that overruled by a prefix match written by someone else.

Fix

Use the dry run, which exists for precisely this type and no other: “Specifying dryRun is valid only when you specify RETENTION_POLICY for policyType”, and it reports decreasedRetentionLogGroupCount, increasedRetentionLogGroupCount, totalMatchedLogGroups and the top log groups impacted. Read the decreased count as the number of log groups about to lose data. And consider NEW_ONLY rather than RETROACTIVE: it “applies the policy only to log groups that are created after you create the policy”, which sets a default for the future without rewriting the past.

2selectionCriteria is a different language for every policy type

The parameter has one name and one type, and what it accepts depends entirely on what you are creating. First, it does not always exist: “Specifying selectionCriteria is valid only when you specify SUBSCRIPTION_FILTER_POLICY, FIELD_INDEX_POLICY, TRANSFORMER_POLICY, or RETENTION_POLICY for policyType.” Data protection takes none — it is all log groups or nothing.

Then each supported type accepts something different:

“If policyType is SUBSCRIPTION_FILTER_POLICY, the only supported selectionCriteria filter is LogGroupName NOT IN []” — exclusion by exact name only. You cannot subscribe a prefix; you can only subscribe everything and name what to leave out.

“If policyType is TRANSFORMER_POLICY, the only supported selectionCriteria filter is LogGroupNamePrefix” — inclusion by prefix only, with no way to exclude.

Field index policies take a prefix or a data source name and type. Retention takes a prefix only. Metric extraction, which cannot use selectionCriteria by the rule above, is documented with its own criteria supporting LogGroupName and LogGroupNamePrefix with IN and NOT IN — the richest of the lot, with “up to 50 values in each IN or NOT IN list” and a limit of one NOT IN policy per account.

The consequence is practical and sharp: whether you can exclude a log group depends on which policy you are writing. #115's recursion loop is prevented by excluding the delivery path's own log groups, and the documentation points there directly — “Using the selectionCriteria parameter with SUBSCRIPTION_FILTER_POLICY is useful to help prevent infinite loops”. That works because subscription filters support NOT IN. A transformer policy, which supports only a prefix, cannot express the same exclusion at all.

Fix

Write the exclusion list first and check it is expressible in the policy type you need, before designing around it. Where a type supports only prefixes, the naming convention of your log groups becomes the access-control mechanism, which is worth knowing before you need to carve one out. And keep selectionCriteria within “up to 25KB in length”, which is generous for prefixes and finite for an explicit name list.

3The counts and the overlap rules are per type, and retention breaks that pattern too

How many policies you can have is not one number. Data protection: “Each account can have only one account-level data protection policy.” Subscription filters: “one account-level subscription filter policy per Region”. Transformers: one for everything, or “as many as 20 account-level transformer policies that are each scoped to a subset of log groups”. Metric extraction: “either one account-level metric extraction policy that applies to all log groups, or up to 5 policies”. Field indexes: 20 by prefix plus another 20 by data source. Retention: “up to 20 account-level retention policies in each Region”.

More consequential than the counts is the overlap rule, which is the same for every type except one. For transformers, field indexes and metric extraction, overlapping scopes are rejected — “no two of them can use the same or overlapping log group name prefixes”. Then:

“Unlike other account-level policy types, retention policies can have overlapping selection criteria. When more than one retention policy applies to a log group, the policy with the longest retention period is used.”

That is a deliberate and well-chosen design — a baseline for the account, longer periods for specific prefixes, and NEVER_EXPIRE “treated as the longest retention period” to exempt what must not be deleted. But it means retention is the one type where you cannot reason about a log group by finding the policy that matches it. You have to find all of them and take the maximum.

Two more retention-only behaviours worth holding: the floor, since “Account-level retention policies require a minimum retention period of 30 days”, and the delay — “A retention policy takes effect 24 hours after you create it”, during which DescribeLogGroups still reports the old period.

Fix

For retention, enumerate every policy that matches a log group rather than the first, and remember the longest wins. For every other type, treat an overlap error as the service telling you two policies disagree about who owns a log group — that is a design question, not a syntax problem.

Architecture

Diagram: CloudWatch Logs PutAccountPolicy creates six kinds of account-level policy with one API call: data protection, subscription filter, field index, transformer, metric extraction and retention. Account-level policies are Region-specific, applying only to log groups in the Region where they are created. The conflict-resolution rule between an account-level policy and a log-group-level policy is different for each type. Data protection policies are cumulative, so any sensitive term specified in either policy is masked. Account-level and log-group-level subscription filters are additive, with up to one account-level and up to five log-group-level filters on a log group. A log-group-level transformer overrides the account-level transformer entirely. For field indexes, a log-group-level policy causes any account-level policy using log group name prefix criteria or no criteria to be ignored for that log group, while account-level policies using data source name and type criteria still apply alongside it. Metric extraction policies are account-level only and have no log-group equivalent. Retention inverts the pattern: an account-level retention policy replaces the retention period set on a log group, including when the log group's own period is longer, so a log group set to ten years under a new ninety-day policy has its older events deleted. Retention is also the only type with a dry run, which reports how many log groups would have decreased, increased or unchanged retention and the top log groups impacted, and the only type whose selection criteria may overlap, where the longest retention period wins and never-expire counts as longest. Selection criteria differ per type: subscription filters support only log group name NOT IN a list, transformers only a log group name prefix, field indexes a prefix or a data source name and type, retention a prefix only, and data protection none at all, while metric extraction supports names and prefixes with IN and NOT IN operators, up to fifty values per list and only one NOT IN policy per account. Policy counts also differ: one data protection policy per account, one subscription filter policy per Region, one transformer policy for everything or up to twenty scoped, up to five metric extraction policies, twenty field index policies by prefix plus twenty by data source, and up to twenty retention policies per Region. Account retention policies require a minimum of thirty days, take effect twenty-four hours after creation, and support a retroactive or new-only enforcement mode. Disabling embedded metric format through a metric extraction policy also disables Container Insights and Application Signals metrics, so the documentation recommends excluding those log groups. The permission to call this API should go only to administrators who manage logging configuration across the account.

Six types, six rules

Policy typeWhen a log-group policy also appliesAccount-level count
Data protection “The two policies are cumulative” — terms from both are masked One per account
Subscription filter “additive” — 1 account-level plus up to 5 log-group-level One per Region
Transformer The log group's own transformer wins; the account one is ignored One for all, or up to 20 scoped
Field index Prefix-scoped account policies ignored; data-source ones still apply 20 by prefix + 20 by data source
Metric extraction No log-group equivalent exists One for all, or up to 5 scoped
Retention The account policy replaces the log group's setting Up to 20 per Region

Reading down the middle column is the whole point of this post. Five rows say the log group keeps what it has, or gets more. The last row says the opposite, and the thing it takes away is data.

What each type lets you select

Policy typeselectionCriteriaCan you exclude a log group?
Data protectionNot supportedNo — all log groups
Subscription filterLogGroupName NOT IN []Yes, by exact name only
TransformerLogGroupNamePrefixNo — inclusion by prefix only
Field indexLogGroupNamePrefix, or DataSourceName + DataSourceTypeNo — inclusion only
Metric extractionLogGroupName / LogGroupNamePrefix with IN and NOT INYes, and only one NOT IN policy per account
RetentionLogGroupNamePrefixNo — but overlapping policies let a longer period win

The asymmetry between the second and third rows is the one that bites. A subscription filter can only be written as everything except these names; a transformer can only be written as these prefixes. Neither can express the other's shape. If your mental model is that an account policy simply takes a scope, both will surprise you, in opposite directions.

Retention's row is subtler. It cannot exclude, but because overlapping is permitted and “the policy with the longest retention period is used”, a second policy with a longer period over a narrower prefix is how you carve an exception out of a baseline — exclusion expressed as precedence rather than as a filter.

The retention lifecycle, which has three timing rules

A floor: “Account-level retention policies require a minimum retention period of 30 days.” You cannot use this API to impose a one-week retention across an account.

A delay: “A retention policy takes effect 24 hours after you create it”, and during that window the previously effective period still governs both DescribeLogGroups and expiration — though a dry run “reports the impact as if the policy were already in effect”. So the console can disagree with your intent for a day, correctly.

A choice about the past: RETROACTIVE “applies the policy to every log group that matches the selection criteria, including log groups that existed before you created the policy”, while NEW_ONLY “applies the policy only to log groups that are created after you create the policy”. This is the single most important field in the request, because it decides whether the policy is a standard for the future or an edit to the past.

The one type that can switch other services off

Metric extraction policies control whether EMF produces metrics, and the warning is explicit: “Creating a policy disables metrics for AWS features that use EMF to create metrics, such as CloudWatch Container Insights and CloudWatch Application Signals.”

With the remedy given as a literal exclusion list to copy — LogGroupNamePrefix NOT IN ["/aws/containerinsights", "/aws/ecs/containerinsights", "/aws/application-signals/data"]. This is the clearest case in the API of a cost-control action silently removing an observability capability two layers away, and #114's note that embedded metrics format is unsupported in the Essentials class is the same capability disappearing for a different reason.

Region scope, and the cross-account inversion

“Account-level policies are Region-specific: a policy applies only to log groups in the Region where you create it. To apply a policy across multiple Regions, create the policy separately in each Region.” Every count above is therefore per Region, and so is every gap.

And the same cross-account inversion #113 found for index policies is stated here too: “If you create an account-level field index policy in a monitoring account in cross-account observability, the policy is applied only to the monitoring account and not to any source accounts.” The account where you run the wide queries is the one where the policy does not reach the data.

Why This Architecture Holds Up

Retention overriding the log group is the only way a retention standard can work

A governance control that any log group owner could opt out of by setting a longer period would not be a control. If the organisation's rule is ninety days, the rule has to beat the local setting. The design is right; what it owes you in exchange is a way to see the consequence first, and that is exactly what the dry run is — available for this type alone, because this is the only type whose application destroys something.

Different conflict rules are the honest answer, not an inconsistency

Masking two sets of sensitive terms is meaningfully cumulative. Running two transformers over one event is not — one of them has to win. Two subscription filters delivering to two destinations is genuinely additive. The rules differ because the underlying operations differ, and a uniform rule would have been wrong for four of the six.

Restricting the permission is the right first control

“You should grant logs:PutAccountPolicy permissions only to administrators who manage logging configuration across the account, not to application teams or individual log group owners.” Given that one call can delete ten years of logs across every matching log group, naming the audience in the API reference is a reasonable place to start.

Key Architecture Decisions

DecisionChoose thisBecause
Any retention policy, ever Dry run first, read decreasedRetentionLogGroupCount It is the only type that deletes, and the only type with a dry run
Setting an account retention standard NEW_ONLY unless you intend to edit history RETROACTIVE reaches log groups that predate the policy
Protecting a log group from a retention baseline A second, longer policy on a narrower prefix Overlap is allowed for retention and the longest period wins
Excluding log groups from a subscription LogGroupName NOT IN [] with exact names It is the only criteria this type accepts, and it prevents recursion loops
Scoping a transformer Design the log group naming convention first Prefix is the only selector, so names are the scope
Disabling EMF to cut metric cost Exclude the Container Insights and Application Signals prefixes Otherwise the policy silently switches those features off
Who holds logs:PutAccountPolicy Logging administrators only AWS says so explicitly, and one call can delete data account-wide
Multi-Region estates Create every policy in every Region, and audit per Region Account-level means account-and-Region

The audit worth running

List every account-level retention policy in every Region, with its enforcement mode. For each, ask which log groups have a longer local setting. That set is the data the policy is deleting or will delete, and no other policy type in this API has an equivalent question.

Check whether any metric extraction policy omits the Container Insights and Application Signals exclusions. If so, those features are off, and nothing about their dashboards will say why.

Compare your subscription filter policy's NOT IN list against your delivery path. This is #115's recursion check, and it lives here because selectionCriteria is where it is expressed.

Confirm the policies exist in every Region you log in. A policy created once, in the Region someone happened to be looking at, covers exactly that Region.

In a monitoring account, check what your field index policies actually reach. They do not apply to source accounts.

Closing Thought

The last four posts have been about monitoring that says nothing when something is wrong. This one is the opposite failure: a mechanism that does exactly what it says, loudly and account-wide, where the risk is that you generalised from the wrong example.

Learn account-level policies through data protection and they are cumulative and safe. Learn them through subscription filters and they are additive. Learn them through transformers and the log group always wins. Every one of those lessons is correct, and every one of them is wrong about retention — which is the type that overrides the log group's own choice and deletes what is older than the new period, including from log groups whose owners deliberately set ten years.

So the rule for this API is to treat policyType as selecting a different product rather than a different mode. Read that type's conflict rule, its selection language and its counts before writing anything. And for retention specifically, run the dry run — it exists for exactly one of the six types, which is AWS telling you which one is different.

Next in this series

Security — CloudWatch Logs data protection policies: what the audit and deidentify blocks each do, why masking applies only to events ingested after the policy exists, who can read through a mask with logs:Unmask, and where the audit findings go.

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent