Business Challenge
The last three posts each ended at the same door. #113's field indexes have an account-level form.
#114's log classes are set per log group but governed in bulk. #115's subscription filters are
account-level or log-group-level, and the exclusions that prevent a recursion loop live in
selectionCriteria. All of it runs through one API:
“Creates an account-level data protection policy, subscription filter policy, field index policy, transformer policy, metric extraction policy, or retention policy that applies to all log groups, a subset of log groups, or a data source name and type combination in the account.”
Six nouns in one sentence, behind one operation, with one policyType parameter. The
shape invites you to learn it once. The documentation then spends several screens explaining that
almost nothing generalises between them.
Every other type resolves a conflict in the log group's favour, or in favour of doing more rather than less: for data protection, “The two policies are cumulative. Any sensitive term specified in either the account-level or the log-group-level policy is masked”; “Account-level and log-group-level subscription filters are additive”; “A log-group-level transformer overrides the account-level transformer”; and for indexes, a log-group policy causes prefix-scoped account policies to be “ignored for that log group”.
Then retention:
“An account-level retention policy replaces the retention period set on a log group, including when the log group's own retention period is longer.”
With AWS supplying the worked example, in bold type on its own page: “For example, if a log group is set to retain log events for 10 years and you create a retention policy that sets a 90-day retention period for that log group, CloudWatch Logs deletes the log events in that log group that are older than 90 days.”
Nothing else in this API removes data. A transformer reshapes new events, a subscription copies them, an index speeds up queries, a data protection policy masks on ingestion. Retention is the one that reaches backwards into what you already have, and it does so by overriding the explicit choice a log group's owner made. A team that set ten years because a regulator asked for ten years has that overruled by a prefix match written by someone else.
FixUse the dry run, which exists for precisely this type and no other: “Specifying dryRun is valid only when you specify RETENTION_POLICY for policyType”, and it reports decreasedRetentionLogGroupCount, increasedRetentionLogGroupCount, totalMatchedLogGroups and the top log groups impacted. Read the decreased count as the number of log groups about to lose data. And consider NEW_ONLY rather than RETROACTIVE: it “applies the policy only to log groups that are created after you create the policy”, which sets a default for the future without rewriting the past.
The parameter has one name and one type, and what it accepts depends entirely on what you are creating. First, it does not always exist: “Specifying selectionCriteria is valid only when you specify SUBSCRIPTION_FILTER_POLICY, FIELD_INDEX_POLICY, TRANSFORMER_POLICY, or RETENTION_POLICY for policyType.” Data protection takes none — it is all log groups or nothing.
Then each supported type accepts something different:
“If policyType is SUBSCRIPTION_FILTER_POLICY, the only supported selectionCriteria filter is LogGroupName NOT IN []” — exclusion by exact name only. You cannot subscribe a prefix; you can only subscribe everything and name what to leave out.
“If policyType is TRANSFORMER_POLICY, the only supported selectionCriteria filter is LogGroupNamePrefix” — inclusion by prefix only, with no way to exclude.
Field index policies take a prefix or a data source name and type. Retention takes a
prefix only. Metric extraction, which cannot use selectionCriteria by the rule above,
is documented with its own criteria supporting LogGroupName and
LogGroupNamePrefix with IN and NOT IN — the richest
of the lot, with “up to 50 values in each IN or NOT IN list” and a limit of
one NOT IN policy per account.
The consequence is practical and sharp: whether you can exclude a log group depends on
which policy you are writing. #115's recursion loop is prevented by excluding the
delivery path's own log groups, and the documentation points there directly —
“Using the selectionCriteria parameter with SUBSCRIPTION_FILTER_POLICY is useful to help
prevent infinite loops”. That works because subscription filters support
NOT IN. A transformer policy, which supports only a prefix, cannot express the same
exclusion at all.
Write the exclusion list first and check it is expressible in the policy type you need, before designing around it. Where a type supports only prefixes, the naming convention of your log groups becomes the access-control mechanism, which is worth knowing before you need to carve one out. And keep selectionCriteria within “up to 25KB in length”, which is generous for prefixes and finite for an explicit name list.
How many policies you can have is not one number. Data protection: “Each account can have only one account-level data protection policy.” Subscription filters: “one account-level subscription filter policy per Region”. Transformers: one for everything, or “as many as 20 account-level transformer policies that are each scoped to a subset of log groups”. Metric extraction: “either one account-level metric extraction policy that applies to all log groups, or up to 5 policies”. Field indexes: 20 by prefix plus another 20 by data source. Retention: “up to 20 account-level retention policies in each Region”.
More consequential than the counts is the overlap rule, which is the same for every type except one. For transformers, field indexes and metric extraction, overlapping scopes are rejected — “no two of them can use the same or overlapping log group name prefixes”. Then:
“Unlike other account-level policy types, retention policies can have overlapping selection criteria. When more than one retention policy applies to a log group, the policy with the longest retention period is used.”
That is a deliberate and well-chosen design — a baseline for the account, longer periods for
specific prefixes, and NEVER_EXPIRE
“treated as the longest retention period” to exempt what must not be deleted.
But it means retention is the one type where you cannot reason about a log group by finding
the policy that matches it. You have to find all of them and take the maximum.
Two more retention-only behaviours worth holding: the floor, since
“Account-level retention policies require a minimum retention period of 30
days”, and the delay —
“A retention policy takes effect 24 hours after you create it”, during which
DescribeLogGroups still reports the old period.
For retention, enumerate every policy that matches a log group rather than the first, and remember the longest wins. For every other type, treat an overlap error as the service telling you two policies disagree about who owns a log group — that is a design question, not a syntax problem.
Architecture
Six types, six rules
| Policy type | When a log-group policy also applies | Account-level count |
|---|---|---|
| Data protection | “The two policies are cumulative” — terms from both are masked | One per account |
| Subscription filter | “additive” — 1 account-level plus up to 5 log-group-level | One per Region |
| Transformer | The log group's own transformer wins; the account one is ignored | One for all, or up to 20 scoped |
| Field index | Prefix-scoped account policies ignored; data-source ones still apply | 20 by prefix + 20 by data source |
| Metric extraction | No log-group equivalent exists | One for all, or up to 5 scoped |
| Retention | The account policy replaces the log group's setting | Up to 20 per Region |
Reading down the middle column is the whole point of this post. Five rows say the log group keeps what it has, or gets more. The last row says the opposite, and the thing it takes away is data.
What each type lets you select
| Policy type | selectionCriteria | Can you exclude a log group? |
|---|---|---|
| Data protection | Not supported | No — all log groups |
| Subscription filter | LogGroupName NOT IN [] | Yes, by exact name only |
| Transformer | LogGroupNamePrefix | No — inclusion by prefix only |
| Field index | LogGroupNamePrefix, or DataSourceName + DataSourceType | No — inclusion only |
| Metric extraction | LogGroupName / LogGroupNamePrefix with IN and NOT IN | Yes, and only one NOT IN policy per account |
| Retention | LogGroupNamePrefix | No — but overlapping policies let a longer period win |
The asymmetry between the second and third rows is the one that bites. A subscription filter can only be written as everything except these names; a transformer can only be written as these prefixes. Neither can express the other's shape. If your mental model is that an account policy simply takes a scope, both will surprise you, in opposite directions.
Retention's row is subtler. It cannot exclude, but because overlapping is permitted and “the policy with the longest retention period is used”, a second policy with a longer period over a narrower prefix is how you carve an exception out of a baseline — exclusion expressed as precedence rather than as a filter.
The retention lifecycle, which has three timing rules
A floor: “Account-level retention policies require a minimum retention period of 30 days.” You cannot use this API to impose a one-week retention across an account.
A delay: “A retention policy takes effect 24 hours after you create
it”, and during that window the previously effective period still governs both
DescribeLogGroups and expiration — though a dry run
“reports the impact as if the policy were already in effect”. So the console can
disagree with your intent for a day, correctly.
A choice about the past: RETROACTIVE
“applies the policy to every log group that matches the selection criteria, including log
groups that existed before you created the policy”, while NEW_ONLY
“applies the policy only to log groups that are created after you create the
policy”. This is the single most important field in the request, because it decides whether
the policy is a standard for the future or an edit to the past.
The one type that can switch other services off
Metric extraction policies control whether EMF produces metrics, and the warning is explicit: “Creating a policy disables metrics for AWS features that use EMF to create metrics, such as CloudWatch Container Insights and CloudWatch Application Signals.”
With the remedy given as a literal exclusion list to copy —
LogGroupNamePrefix NOT IN ["/aws/containerinsights", "/aws/ecs/containerinsights",
"/aws/application-signals/data"]. This is the clearest case in the API of a cost-control action
silently removing an observability capability two layers away, and #114's note that embedded metrics
format is unsupported in the Essentials class is the same capability disappearing for a different
reason.
Region scope, and the cross-account inversion
“Account-level policies are Region-specific: a policy applies only to log groups in the Region where you create it. To apply a policy across multiple Regions, create the policy separately in each Region.” Every count above is therefore per Region, and so is every gap.
And the same cross-account inversion #113 found for index policies is stated here too: “If you create an account-level field index policy in a monitoring account in cross-account observability, the policy is applied only to the monitoring account and not to any source accounts.” The account where you run the wide queries is the one where the policy does not reach the data.
Why This Architecture Holds Up
Retention overriding the log group is the only way a retention standard can work
A governance control that any log group owner could opt out of by setting a longer period would not be a control. If the organisation's rule is ninety days, the rule has to beat the local setting. The design is right; what it owes you in exchange is a way to see the consequence first, and that is exactly what the dry run is — available for this type alone, because this is the only type whose application destroys something.
Different conflict rules are the honest answer, not an inconsistency
Masking two sets of sensitive terms is meaningfully cumulative. Running two transformers over one event is not — one of them has to win. Two subscription filters delivering to two destinations is genuinely additive. The rules differ because the underlying operations differ, and a uniform rule would have been wrong for four of the six.
Restricting the permission is the right first control
“You should grant logs:PutAccountPolicy permissions only to administrators who manage logging configuration across the account, not to application teams or individual log group owners.” Given that one call can delete ten years of logs across every matching log group, naming the audience in the API reference is a reasonable place to start.
Key Architecture Decisions
| Decision | Choose this | Because |
|---|---|---|
| Any retention policy, ever | Dry run first, read decreasedRetentionLogGroupCount |
It is the only type that deletes, and the only type with a dry run |
| Setting an account retention standard | NEW_ONLY unless you intend to edit history |
RETROACTIVE reaches log groups that predate the policy |
| Protecting a log group from a retention baseline | A second, longer policy on a narrower prefix | Overlap is allowed for retention and the longest period wins |
| Excluding log groups from a subscription | LogGroupName NOT IN [] with exact names |
It is the only criteria this type accepts, and it prevents recursion loops |
| Scoping a transformer | Design the log group naming convention first | Prefix is the only selector, so names are the scope |
| Disabling EMF to cut metric cost | Exclude the Container Insights and Application Signals prefixes | Otherwise the policy silently switches those features off |
Who holds logs:PutAccountPolicy |
Logging administrators only | AWS says so explicitly, and one call can delete data account-wide |
| Multi-Region estates | Create every policy in every Region, and audit per Region | Account-level means account-and-Region |
The audit worth running
List every account-level retention policy in every Region, with its enforcement mode. For each, ask which log groups have a longer local setting. That set is the data the policy is deleting or will delete, and no other policy type in this API has an equivalent question.
Check whether any metric extraction policy omits the Container Insights and Application Signals exclusions. If so, those features are off, and nothing about their dashboards will say why.
Compare your subscription filter policy's NOT IN list against your delivery
path. This is #115's recursion check, and it lives here because
selectionCriteria is where it is expressed.
Confirm the policies exist in every Region you log in. A policy created once, in the Region someone happened to be looking at, covers exactly that Region.
In a monitoring account, check what your field index policies actually reach. They do not apply to source accounts.
Closing Thought
The last four posts have been about monitoring that says nothing when something is wrong. This one is the opposite failure: a mechanism that does exactly what it says, loudly and account-wide, where the risk is that you generalised from the wrong example.
Learn account-level policies through data protection and they are cumulative and safe. Learn them through subscription filters and they are additive. Learn them through transformers and the log group always wins. Every one of those lessons is correct, and every one of them is wrong about retention — which is the type that overrides the log group's own choice and deletes what is older than the new period, including from log groups whose owners deliberately set ten years.
So the rule for this API is to treat policyType as selecting a different product rather
than a different mode. Read that type's conflict rule, its selection language and its counts before
writing anything. And for retention specifically, run the dry run — it exists for exactly one of
the six types, which is AWS telling you which one is different.
Security — CloudWatch Logs data protection policies: what the audit and deidentify blocks each do, why masking applies only to events ingested after the policy exists, who can read through a mask with logs:Unmask, and where the audit findings go.
Official AWS Reference
- PutAccountPolicy — the six policy types and the conflict-resolution rule for each, the retention policy that replaces a log group's own period and deletes older events, the per-type
selectionCriteriasupport and the 25KB limit, every policy count including the 20 transformer, 20 plus 20 field index, 5 metric extraction and 20 retention policies, the 30-day minimum and 24-hour delay and theRETROACTIVEandNEW_ONLYenforcement modes, the dry run available only for retention, the EMF warning about Container Insights and Application Signals, the Region scoping, the monitoring-account limitation, and the guidance on who should hold the permission - Log recursion prevention — further reading on why a subscription filter policy's
NOT INexclusions matter, covered in full in #115; no claims in this post are drawn from it - Create field indexes to improve query performance and reduce scan volume — further reading on the log-group-level form of index policies, covered in full in #113; no claims in this post are drawn from it
- Protect sensitive log data with masking — further reading on data protection policies and the
logs:Unmaskpermission; no claims in this post are drawn from it - DryRunResult — further reading on the fields a retention dry run returns; no claims in this post are drawn from it
Comments