Normalised is not the same as stable

Business Challenge

#43 put the security services in one account and #51 got findings out of one of them as a file. Security Lake is the version of that question asked about everything at once: “automatically centralize security data from AWS environments, SaaS providers, on premises, cloud sources, and third-party sources into a purpose-built data lake that's stored in your AWS account”.

The conversion is the product: “Security Lake converts ingested data into Apache Parquet format and a standard open-source schema called the Open Cybersecurity Schema Framework (OCSF).” One shape, many sources, no post-processing.

1The schema has versions, and they disagree

AWS publishes the source identification twice — a table for “Version 1” at 1.0.0-rc.2, and one for “Version 2” at 1.1.0. Most rows agree. One does not, and it is the one carrying your findings:

Security Hub CSPM	`class_name`
Version 1 — `1.0.0-rc.2`	“Security Finding”
Version 2 — `1.1.0`	“Vulnerability Finding, Compliance Finding, or Detection Finding”

One class became three. A dashboard that filters class_name = 'Security Finding' was correct when it was written and matches nothing afterwards. SQL does not warn you that a predicate has stopped matching — an empty result set is a valid answer to a well-formed query.

This is the same failure shape as #75, where a control passed because its population was empty. A zero that means "none matched the filter" and a zero that means "nothing happened" are indistinguishable from the number.

Fix

Alert on a query returning zero rows where it previously returned many. For a security dataset, an unexplained drop to zero is the signal, not the absence of one.

2Two subscriber access types, and only one can be constrained

"Subscriber" is one word for two quite different grants, distinguished by the accessTypes parameter.

S3 — “Subscribers can directly access the S3 objects and receive notifications of new objects through a subscription endpoint or by polling an Amazon Simple Queue Service (Amazon SQS) queue.” Whole objects, as written.

LAKEFORMATION — “These subscribers directly query AWS Lake Formation tables in your S3 bucket with services like Amazon Athena.” Rows through a table, governed by Lake Formation.

The difference decides what you can withhold. A Parquet object contains every column of every record in it; handing one over is handing over all of it. A Lake Formation table is the layer where column-level and row-level controls live. Both are "a subscriber", and only the second gives you anywhere to express "this team sees these fields".

Fix

Default to LAKEFORMATION. Reach for S3 when the consumer genuinely needs the raw objects — a third-party SIEM ingesting files — and know you have granted the whole record when you do.

3The normalisation is for AWS sources; everything else is your job

The qualifier appears in both descriptions and is easy to read past: “Security Lake automatically converts logs and events that come from natively-supported AWS services to the OCSF schema.”

And for everything else: “Logs and events that are written to Security Lake from custom sources must adhere to the OCSF schema and an Apache Parquet format.”

That is a requirement on you, not a service that runs for you. The promise of one schema across “AWS environments, SaaS providers, on premises” holds, and the conversion work for the non-AWS half sits upstream in whatever you build to land the data. Security Lake stores what you give it and validates the shape; it does not translate it.

Fix

Scope a custom source as a transformation project with a schema-version dependency, not as a connector. It is the OCSF version question again, and this time you own both ends of it.

Architecture

Collection you configure, conversion that applies to some of it, and two doors out.

Diagram: what Amazon Security Lake converts, what a subscriber actually receives under each of its two access types, and what the OCSF version tables imply for queries. Security Lake is a fully managed security data lake service used to centralize security data from AWS environments, SaaS providers, on premises, cloud sources and third-party sources into a purpose-built data lake stored in the customer's own AWS account, backed by Amazon S3 buckets with the customer retaining ownership of the data, and with one bucket per AWS Region. It converts ingested data into Apache Parquet format and the Open Cybersecurity Schema Framework, or OCSF, automatically partitioning incoming data from natively supported AWS services. The crucial qualifier, repeated in the documentation, is that automatic conversion to OCSF applies to natively-supported AWS services only: logs and events written from custom sources must themselves already adhere to the OCSF schema and to Apache Parquet format, so the transformation work for non-AWS sources sits upstream with the customer and Security Lake validates shape rather than translating. On the schema itself, OCSF provides a standard schema for common security events and explicitly defines versioning criteria to facilitate schema evolution, and AWS publishes the source identification mapping twice: a Version 1 table at metadata version 1.0.0-rc.2 and a Version 2 table at 1.1.0. Most rows agree between them but the Security Hub CSPM row does not: in Version 1 its class_name is Security Finding, while in Version 2 the class_name is Vulnerability Finding, Compliance Finding, or Detection Finding. One class became three, so a dashboard or query filtering on class_name equals Security Finding was correct when written and matches nothing against Version 2 data, and SQL does not warn that a predicate stopped matching because an empty result set is a valid answer to a well-formed query. Other mappings are recorded for reference: VPC Flow Logs map to class_name Network Activity, Route 53 Resolver Query Logs to DNS Activity, and CloudTrail Management Events to API Activity, Authentication, or Account Change. A further subtlety is that OCSF is a convention on read rather than an enforcement: subscribers can treat the logs and events as generic Parquet records, or apply the OCSF schema event class to more accurately interpret the information in a record. On consumption, the word subscriber covers two materially different grants distinguished by the accessTypes parameter of the CreateSubscriber API. The S3 access type gives data access: subscribers are notified of new objects as data is written to the S3 bucket, by default through an HTTPS endpoint that the subscriber provides or alternatively by polling an Amazon SQS queue, and those subscribers directly access the S3 objects themselves. The LAKEFORMATION access type gives query access: subscribers directly query AWS Lake Formation tables in the customer's S3 bucket using services like Amazon Athena, which is the primary query engine, though Amazon Redshift Spectrum and Spark SQL also work because they integrate with the AWS Glue Data Catalog. The difference determines what can be withheld, because a Parquet object contains every column of every record within it so handing one over hands over all of it, whereas a Lake Formation table is the layer at which column-level and row-level controls can be expressed. Both are called subscribers and only the second provides anywhere to say that a given team sees only certain fields. Across both types, subscribers may consume data only from the sources and in the AWS Regions that the customer specifies, and Security Lake automatically creates and exchanges the credentials needed between Security Lake and the subscriber. Finally, rollup Regions can be designated to consolidate security log and event data from multiple Regions, which AWS notes can help comply with data residency compliance requirements. — Two access types, two very different grants. Only one has a place to express a restriction.

What conversion actually buys, and where it stops

Two things happen to a natively supported AWS source, and they are worth separating because only one of them is the schema:

Transformation	What it gives you
“automatically partitions incoming data” and converts to Parquet	Columnar storage and partition pruning — this is what makes the queries cheap
Transforms to the OCSF schema	Field names that mean the same thing across sources — this is what makes the queries joinable

The Parquet half is unambiguous and durable. The OCSF half is the one with a version number on it, and AWS says so in the definition rather than in a footnote: OCSF “provides a standard schema for common security events, defines versioning criteria to facilitate schema evolution, and includes a self-governance process”.

Schema evolution is a feature. It is also the thing your queries are pinned to.

OCSF is applied on write and optional on read

“Subscribers can treat the logs and events as generic Parquet records or apply the OCSF schema event class to more accurately interpret the information contained in a record.”

So the schema is a convention the reader opts into, not a contract the storage enforces. A subscriber reading columns positionally gets data; a subscriber applying the event class gets meaning. Both work, and only one notices when the class changes underneath it.

The event class is the join key, conceptually

“Logs and events from a given Security Lake source match a specific event class defined in OCSF. DNS Activity, SSH Activity, and Authentication are examples of event classes.”

That is the payoff of the whole exercise. Route 53 Resolver query logs arrive as “DNS Activity”, VPC Flow Logs as “Network Activity”, and CloudTrail management events as “API Activity, Authentication, or Account Change” depending on what the event was. Ask "what did this principal do" across three services and the class and field names line up.

Note the shape of the CloudTrail row though: a single source maps to three classes depending on the event. So "one source, one class" is already not true before any version change — which means a query written against CloudTrail has to know which class it wants, and the Security Hub change in Version 2 is the same pattern arriving somewhere it previously did not apply.

Where the data sits, and who owns it

“The data lake is backed by Amazon Simple Storage Service (Amazon S3) buckets, and you retain ownership over your data”, stored “in an Amazon Simple Storage Service (Amazon S3) bucket (one bucket per AWS Region) in your AWS account”.

One bucket per Region is the detail with consequences. It makes the Regional boundary a storage boundary, which is why the rollup feature exists: “you can also designate rollup Regions to consolidate security log and event data from multiple Regions… This can help you comply with data residency compliance requirements.”

Read that the other way round and it is a constraint as much as a feature: consolidating into a rollup Region is a deliberate movement of security data across a Regional boundary, and the same mechanism that helps one residency regime can breach another.

Why This Architecture Holds Up

The credential exchange is handled, which removes a real source of error

“Security Lake automatically creates and exchanges the credentials needed between Security Lake and the subscriber.”

Cross-account access to a log bucket is exactly the kind of thing that gets built once with a wildcard and never revisited. Having the service mint and rotate the relationship removes the most common way these integrations go wrong, and it is a meaningful part of the case for Security Lake over a bucket you share yourself.

Subscriber scoping is per source and per Region

“Subscribers may consume data only from the sources, and in the AWS Regions, that you specify.”

Two axes, both enforced by the service rather than by a policy you write. That is a better default than the usual arrangement, where a consumer is pointed at a bucket prefix and trusted to stay in it.

It also composes with the access-type choice in a useful way: source and Region scoping applies to both S3 and LAKEFORMATION subscribers, so even a raw-object subscriber is bounded. What it does not give you is anything finer — within a permitted source and Region, an S3 subscriber has the whole record.

The version question is the one to answer before you build dashboards

AWS gives the mappings for both versions rather than only the current one, which is the honest thing to do and also tells you this has already happened once. The Security Hub CSPM row is the warning:

“Security Finding” under 1.0.0-rc.2 — and under 1.1.0, “Vulnerability Finding, Compliance Finding, or Detection Finding”, with the product and vendor fields changing from fixed strings to “Matches AWS Security Finding Format (ASFF) ProductName value”.

So both the class and the identifying fields moved. Anything built on the Version 1 shape — a saved Athena query, a QuickSight dataset, a detection rule in a third-party tool — needs checking against the version your data actually carries, and metadata.version is the field that tells you.

Custom sources are a contract, and contracts are the right model

Requiring custom sources to “adhere to the OCSF schema and an Apache Parquet format” is more work than a service that ingests anything and converts it. It is also the only version that keeps the promise: a lake where some records follow the schema and others do not is a lake where no query is trustworthy.

The cost lands in planning rather than in operation. A custom source is a piece of software you maintain against a versioned schema, and the version it targets is a dependency to track like any other.

Key Architecture Decisions

Decision	Choice	Reasoning
Subscriber access type	`LAKEFORMATION` unless raw objects are genuinely needed	It is the only layer where a column or row restriction can be expressed.
Granting `S3` access	Treat it as granting the whole record	Subscribers “directly access the S3 objects”, and an object carries every column.
Pinning queries	Filter on `metadata.version`, or handle both	Two published mappings, and the Security Hub class changed between them.
Dashboards and saved queries	Alert when a result count drops to zero	A stale predicate returns an empty set, not an error.
CloudTrail queries	Expect three classes from one source	“API Activity, Authentication, or Account Change” depending on the event.
Custom sources	Plan as a transformation project	They “must adhere to the OCSF schema and an Apache Parquet format” on arrival.
Non-AWS data	Do not assume conversion	Automatic OCSF transformation is for “natively-supported AWS services”.
Rollup Regions	Decide against your residency rules, in both directions	The same consolidation that satisfies one regime moves data out of another.
Subscriber scope	Set sources and Regions explicitly	The service enforces both, so use them rather than relying on convention.
Reading records	Apply the event class rather than reading positionally	Generic Parquet reads work and will not notice a schema change.

The audit worth running

List your saved queries, dashboards and detection rules that reference an OCSF class_name, and check each against the metadata.version actually present in your lake. Every one written before the Version 2 mapping is a candidate for silently returning nothing.

Then list your subscribers by accessTypes. Each S3 entry is a consumer holding complete records for the sources and Regions it is scoped to, and that is worth knowing deliberately rather than discovering during a review.

Closing Thought

Security Lake does the thing it says. The Parquet conversion makes the data cheap to query, the OCSF transformation makes it joinable across sources, the credential exchange removes a class of misconfiguration, and source-and-Region scoping is enforced rather than advised. For centralising security telemetry it is a better answer than a bucket and a convention.

What the word "normalised" invites you to assume is that the shape is settled. It is not, and AWS does not claim it is — the definition says OCSF “defines versioning criteria to facilitate schema evolution”, and the two published tables are that evolution having already happened once. A standard schema removes the problem of sources disagreeing with each other. It does not remove the problem of the schema disagreeing with itself a year apart.

Which is the same lesson as the four posts before it, arriving from a new direction. #78 through #81 were each about a fact asserted at one moment and read at another. Here the fact is the schema itself, the moment is the OCSF version, and the thing that reads it is a query you wrote and have not looked at since. An empty table is the quietest way a system has of telling you something changed.

Next in this series

Security & Identity — AWS Audit Manager: what an assessment actually collects, why evidence is gathered continuously but assessed against a framework version, and what happens to an assessment when the framework underneath it is updated.

AWS Architecture Series #82 — Normalised is not the same as stable