Homeβ€Ί Blogβ€Ί AWS Architecture Series #117 β€” Masking hides the data on the way out; one permission puts it back…
AWS Architecture AWS Architecture Series

AWS Architecture Series #117 β€” Masking hides the data on the way out; one permission puts it back

A CloudWatch Logs data protection policy masks sensitive data at all egress points - Logs Insights, metric filters, subscription filters - and only holders of logs:Unmask can see through it. That is a read-time control, not redaction: the mask is applied on the way out, which is why a single IAM permission reverses it. Masking also only applies to events ingested after the policy exists, so turning it on does not clean history. And the audit half of the policy is optional, so a policy can be masking data while keeping no record of what it found.

Verified against current vendor documentation on 18 November 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

#116 covered the six account-level policy types and promised a closer look at the one with the strongest-sounding name. Data protection policies “let you audit and mask sensitive data that appears in log events ingested by the log groups in your account”, across a long list of managed identifiers — credentials, financial data, PII, PHI, device identifiers.

The word that does the work is mask, and it is doing something narrower than most readers hear. Masking is not redaction, and the documentation says so clearly enough if you read two sentences together rather than one.

1The mask is applied at egress, so one IAM permission removes it

“When you create a data protection policy, then by default, sensitive data that matches the data identifiers you've selected is masked at all egress points, including CloudWatch Logs Insights, metric filters, and subscription filters.”

And immediately after: “Only users who have the logs:Unmask IAM permission can view unmasked data.” Concretely, that user “can use a GetLogEvents or FilterLogEvents operation with the unmask parameter set to true to view the unmasked log events”.

Put those together and the architecture is unambiguous. The masking happens on the way out, which means the unmasked value is still there to be returned when someone asks with the right permission and a parameter. This is a presentation control over retained data, not a destruction control. Both are legitimate designs; they answer different questions, and a compliance conversation that treats the first as the second is building on sand.

The egress list is the reassuring half, and it is genuinely broad: Logs Insights, metric filters and subscription filters are the three ways data leaves a log group, and all three mask. The copy that reaches your SIEM through #115's subscription filter is masked. What is not protected is the data at rest, from anyone holding logs:Unmask.

Fix

Treat logs:Unmask as a tightly governed permission: granted to a named role, never to a human group. Monitor supported CloudTrail activity associated with access to CloudWatch Logs, and verify which unmask operations and request parameters are captured before relying on CloudTrail alerts as evidence of every unmasked read — IAM authorization and audit coverage are different controls. The data protection policy decides what is hidden; that permission decides who the hiding does not apply to, and it is the shorter list to audit.

2It starts when you create it, and the audit half is optional

The timing rule is in an Important callout, and it is the same shape as #113's field indexes:

“Sensitive data is detected and masked when it is ingested into the log group. When you set a data protection policy, log events ingested to the log group before that time are not masked.”

So a policy created in response to an incident does nothing about the incident. The credentials already in the log group stay visible to everyone who could already read them, and the only remedies are the ones that were always available: delete the log stream, shorten retention, or rotate what leaked. For an account-level policy there is also a propagation delay — “It might take up to 5 minutes before sensitive data in existing log groups begins to be masked”.

The second half of this is the policy's own structure. A data protection policy is two JSON blocks: an Audit action and a Deidentify action, where “The Operation property with the Deidentify action is what actually masks the data” and the identifier lists “must exactly match” between them. The audit block is mandatory — it is “required to find the sensitive data terms” — and it must contain a FindingsDestination object.

But: “You can optionally use that FindingsDestination object to list one or more destinations to send audit findings to.” An empty destination list is valid. So the common outcome is a policy that masks correctly and keeps no durable record of what it found, leaving only the LogEventsWithFindings metric — a count, with no indication of which log group, which identifier, or how often.

Fix

Configure a findings destination when you create the policy, not later, and note that destinations “must already exist” before the policy will accept them. If the destination is an S3 bucket with SSE-KMS, the extra step is real: “you must update the key policy for your KMS key so that the log delivery account can write to your S3 bucket”, whereas SSE-S3 needs nothing. And alarm on LogEventsWithFindings regardless, since it is free — “vended metrics and are free of charge”.

3Detection is probabilistic, and the metric counts events rather than findings

Two statements set expectations correctly, and both are easy to skim past.

On the mechanism: “CloudWatch Logs data protection allows you to leverage pattern matching and machine learning models to detect sensitive data”, and — the important one — “For some types of managed data identifiers, the detection depends on also finding certain keywords in proximity with the sensitive data.”

That is the difference between a regex and a detector. A driver's licence number in a log line with no nearby word suggesting it is a driver's licence may not be detected at all. The design is sound — context-free matching on a nine-digit number would mask half your request IDs — but it means coverage is a function of how your application happens to phrase its log messages, which nobody chose deliberately.

On measurement: “Even though this log event has two email addresses, the value of the LogEventsWithFindings metric is incremented only by one, because that metric counts the number of log events that contain sensitive data, not the number of occurrences of sensitive data.”

So the metric is a lower bound on exposure, and a bad one if your leak is a single log event containing a thousand records. A flat line at one per minute can be one address or a bulk dump.

Fix

Write a custom data identifier for anything whose format you control — internal account numbers, customer references, tenant IDs — rather than hoping a managed identifier infers it: “You can also use custom data identifiers to create data identifiers tailored to your specific use case.” And treat LogEventsWithFindings as a detector of whether rather than how much; the findings reports are where the counts live.

Architecture

Diagram: A CloudWatch Logs data protection policy audits and masks sensitive data in log events. Masking is applied at all egress points, including CloudWatch Logs Insights, metric filters and subscription filters, and only users who have the logs:Unmask IAM permission can view unmasked data, using the unmask parameter on GetLogEvents or FilterLogEvents. That makes the control a read-time transformation rather than redaction: the unmasked value is still retained and one IAM permission returns it, so logs:Unmask is the real control boundary. Masking applies only from the moment the policy exists, because sensitive data is detected and masked when it is ingested into the log group, and log events ingested before that time are not masked, with an account-level policy taking up to five minutes to begin masking in existing log groups. The policy itself is two JSON blocks whose data identifier arrays must exactly match: an Audit action, which is required to find the sensitive data terms and must contain a FindingsDestination object, and a Deidentify action, which is what actually masks the data and must contain an empty MaskConfig object. The findings destination list is optional, so a policy can mask correctly while keeping no durable record of what it found. Audit findings are written one report per log event containing sensitive data, and they contain the log group ARN, the identifier name, a count of occurrences in that event, and the start and end character offsets of each occurrence, but never the sensitive value itself. Destinations must already exist, and an S3 bucket protected with SSE-KMS requires a key policy update so the log delivery account can write to it, while SSE-S3 requires nothing. Detection uses pattern matching and machine learning models, and for some managed data identifiers detection depends on also finding certain keywords in proximity with the sensitive data, so coverage depends on how log messages are phrased; custom data identifiers exist for formats you control. The LogEventsWithFindings metric is emitted in the AWS slash Logs namespace, is free of charge, and counts the number of log events containing sensitive data rather than the number of occurrences, so it is a lower bound on exposure. Account-level and log-group-level policies are cumulative, with all identifiers from either policy audited and masked, each log group may have only one log-group-level policy, the policy document limit is 30,720 characters, and masking is supported in both the Standard and Essentials log classes.

What the control actually does

QuestionAnswer
Is the sensitive value removed from the log group? No — masking is applied “at all egress points”
Who can see through it? Holders of logs:Unmask, via the unmask parameter
Does it cover data already in the log group? No — events ingested before the policy “are not masked”
Does it cover what leaves via a subscription filter? Yes — subscription filters are an egress point
Does it cover Logs Insights queries? Yes
Is there a record of what was found? Only if you configured a findings destination — it is optional
Does it work in the Essentials log class? Yes — “both the Standard and Essentials” classes

That last row is worth noting against #114. Almost everything else in CloudWatch Logs is Standard-only — field indexing, metric filters, subscription filters, Live Tail. Masking works in the cheap class, which makes the consolidation tier a safer place to put logs you have not fully vetted than its feature table would suggest.

The two blocks, and why they must agree

“The first block must include both a DataIdentifer array and an Operation property with an Audit action”; “The second block must include both a DataIdentifer array and an Operation property with an Deidentify action. The DataIdentifer array must exactly match the DataIdentifer array in the first block of the policy.” And the deidentify block is mechanically fixed: it “must contain the "MaskConfig": {} object. The "MaskConfig": {} object must be empty.”

The requirement that the arrays match exactly is the service refusing to let you audit one set of identifiers and mask a different one. That could only produce two outcomes, both bad: data masked with no finding recorded, or a finding recorded for data still visible. Forcing the two lists to be identical removes the whole class of mistake, at the cost of a policy document that looks redundant and invites someone to simplify it.

What a finding tells you, and what it refuses to

“CloudWatch Logs writes one findings report for each log event that contains sensitive data”, containing the log group ARN, the identifier “name”, a “count” of occurrences in that event, and “The start and end fields show where in the log event, by character count, each occurrence of the sensitive data appears.”

What it does not contain is the value. A finding says an email address occupied characters 13 to 26 of this event, never which address. That is the right call and not an obvious one: a findings pipeline that carried the sensitive values would recreate the exposure in a second system, usually a less protected one. Offsets and counts are enough to triage and fix a logging statement without copying the secret into S3.

The one thing to design around is that offsets are a weak side channel in aggregate — stable offsets across many events reveal the shape of your log lines, not their contents. Negligible for most, worth a thought if findings go somewhere broadly readable.

How account and log-group policies combine

From #116's conflict table, this is the cumulative case: “If you create a data protection policy for your entire account and you also create a policy for a single log group, both policies apply to that log group. All managed data identifiers that are specified in either policy are audited and masked in that log group.”

With “Each log group can have only one log group-level data protection policy” and a document limit of “30,720 characters”. Cumulative is the right default for a protective control — a team can add identifiers for their own data without being able to subtract the organisation's. There is no exclusion mechanism here at all, which #116 showed is unusual among the six policy types and, for this one, correct.

Why This Architecture Holds Up

Masking at egress rather than at rest is the defensible choice

Destroying the original at ingestion would make the log group the only copy and the detection irreversible: a false positive would silently corrupt an audit trail, with no way to recover what was overwritten. Masking on read keeps the record intact and makes the exception auditable through one permission. The honest framing is that this control reduces casual exposure across every read path, and does not claim the data is gone.

Keeping values out of findings is a genuinely good decision

The obvious design is for an audit finding to say what it found. Resisting that keeps the sensitive-data blast radius inside the log group instead of fanning it out to every findings destination, and offsets plus counts are sufficient for the actual task — locating the logging statement that emitted it.

Cumulative with no exclusions is right for a protective policy

Every other account-level policy type in #116 has some way to scope or exclude. This one does not, and a protective baseline that a log group owner could narrow would not be a baseline. Additive-only is the correct asymmetry.

Key Architecture Decisions

DecisionChoose thisBecause
The real control boundary Grant logs:Unmask to a named role and govern it tightly Masking is applied at egress and that permission reverses it
Findings destination Configure one at creation It is optional, and without it the only evidence is an event count
Destination encryption SSE-S3 unless you need SSE-KMS SSE-KMS needs a key policy update for the log delivery account; SSE-S3 needs nothing
Responding to a leak already in logs Delete, shorten retention, or rotate — not a policy Events ingested before the policy are never masked
Identifiers for your own formats Custom data identifiers Managed detection can depend on nearby keywords, which your log phrasing decides
Measuring exposure Findings reports, with the metric as the trigger LogEventsWithFindings counts events, not occurrences
Alarming Alarm on the metric from day one It is free, and it is the only always-present signal
Scope Account-level baseline, log-group policies to add The two are cumulative and there is no exclusion mechanism

The audit worth running

List every principal with logs:Unmask. That list is the set of identities for whom your data protection policy does not exist. If it is longer than a handful of tightly governed roles, the policy is decoration.

Check whether each data protection policy has a findings destination. A policy masking without recording is the common configuration and the one that leaves you unable to answer what leaked, when, or from where.

Compare policy creation dates against log group retention. Any log group whose retention reaches back before its policy has an unmasked window still being served, and the length of it is arithmetic you can do today.

Test detection with a synthetic event in the format your application actually logs. Because some identifiers need nearby keywords, the only reliable coverage check is writing a line that looks like your real lines and querying for it.

Establish what your trail actually captures. Monitor supported CloudTrail activity associated with access to CloudWatch Logs, and verify which unmask operations and request parameters are recorded before treating CloudTrail as evidence of every unmasked read. The permission is the boundary; whether each exercise of it is individually recorded is a separate question, and the documentation cited here does not answer it.

Closing Thought

Five posts ago this series was looking at alarms that say nothing. The thread since has been about controls whose name is broader than their guarantee: an index that only covers what arrived after it, a class whose saving is only at ingestion, a subscription whose permissions failure skips the logs, an API whose retention policy deletes.

Data protection policies are the most sharply named of them. Mask sounds terminal, and the control is genuinely strong across every path data takes out of a log group — Logs Insights, metric filters, subscription filters, all of it. But the data is still there, one IAM permission and one request parameter away, which is not a flaw so much as the thing you must know in order to describe the control accurately to an auditor.

So the rule is: the policy decides what is hidden, and logs:Unmask decides who it is hidden from. Govern the second as carefully as you configure the first — and keep the two questions apart, because who is authorised to unmask and what your trail records about unmasking are different controls with different coverage. Record what the audit block finds rather than leaving the destination empty, and remember that none of it reaches backwards to the credential that was logged last Tuesday.

Next in this series

Security — CloudWatch Logs custom data identifiers: what a custom pattern can and cannot express, the documented constraints on it, and why a detector that depends on keywords in proximity behaves differently from a pattern match in ways that decide your coverage.

Official AWS Reference

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent