Lake Formation and Governed Access

Business Challenge

A retail analytics platform has been running for four years. Roughly 900 tables sit across 40 Glue databases, all of it Parquet on S3, queried by Athena from the BI tool, by EMR from the data science team, and by Redshift Spectrum from a warehouse that a finance team owns. Access is decided the way it was decided on day one: one IAM role per consuming team, and a bucket policy that lists those roles.

That worked until the requirements stopped being expressible in it. Legal wants two columns on the customer table — date of birth and national ID — visible only to a named group of six people. The EU entity wants its analysts to see EU rows and nothing else, from the same tables everyone else queries. A partner account needs three tables, and only three, without being handed a role in the platform account.

None of that is a prefix. Columns and rows are not directories, and no arrangement of S3 paths makes them one without physically forking the data into copies that then have to be kept in step. The team does the correct thing: they install Lake Formation, register the bucket, write the grants.

Nothing changes. Every analyst can still read every column. No error appears anywhere.

There are two reasons, and both of them are the default state of the system rather than a mistake anyone made. The first is that Lake Formation sets Super on all databases and tables in the Data Catalog to a group called IAMAllowedPrincipals by default, which means access is still decided entirely by the IAM policies that were already there. The second is deeper and does not go away when you revoke that grant: Lake Formation is not a gate in front of S3. It is an authorisation plane that an engine consults voluntarily, and the EMR job reading s3://lake/customer/ with an instance profile is not consulting anything.

The failure that produces no signal

A denied Lake Formation request appears in CloudTrail. A request that was never made does not. When a principal reads the objects directly with s3:GetObject, there is no metadata call, no permission evaluation, and no filtering — so the grant you wrote is not violated, it is simply never consulted. Every audit built on "show me the denials" reports a clean lake.

Architecture

The whole design follows from one mechanism, so it is worth stating precisely before anything else. AWS calls it credential vending.

Diagram: how Lake Formation enforces access, showing the four-step credential vending path from principal to integrated engine to Data Catalog to Lake Formation to S3, the direct S3 path that consults none of it, the three states a Data Catalog resource can be in as an AWS Glue resource, a Lake Formation resource or a hybrid resource, and the arithmetic that makes tag-based access control scale where named-resource grants do not

Four steps, and the one that decides everything

A principal submits a query to an integrated engine — Athena, EMR, Glue or Redshift Spectrum. The engine asks the Data Catalog for the table. The Data Catalog checks that principal's Lake Formation permissions and returns only the metadata they are allowed to see. Then the catalog tells the engine whether the underlying location is registered with Lake Formation, and if it is, the engine requests temporary credentials. AWS's description of the last step is the load-bearing sentence:

"If the user is authorized to access the table, Lake Formation provides temporary access to the integrated analytical engine. Using the temporary access, the analytical engine fetches the data from Amazon S3, and performs necessary filtering such as column, row, or cell filtering."

Read that again with an architect's eye. The engine does the filtering. Lake Formation never sees a row. It hands out short-lived credentials and trusts the engine to restrict what it returns — which AWS states outright elsewhere: "It is the responsibility of the integrated analytics service to apply the column filtering when processing a query."

And the fall-through case is documented just as plainly: "If the table is not managed by Lake Formation, the second call from the analytic engine is made directly to Amazon S3." The bucket policy and the IAM policy decide, and Lake Formation is not in the conversation at all.

Two permission planes, and they are not the same plane

Lake Formation implements permissions on two levels, and conflating them is the most common source of "I granted SELECT and it still says access denied":

Metadata permissions

Grants on Data Catalog objects — catalogs, databases, tables, views. SELECT, INSERT, DELETE, ALTER, DROP, DESCRIBE, CREATE_TABLE, CREATE_DATABASE, Super. These control what a principal can see and do in the catalog, and they are what most people mean when they say "Lake Formation permissions".

Storage access

Managed on your behalf for S3 locations you registered, via a role that Lake Formation assumes. This is the plane that produces the temporary credentials. If the location was never registered, this plane does not exist for that data and the metadata grants govern the catalog entry while the objects stay wide open.

There is a third grant that belongs to neither and confuses everyone the first time: DATA_LOCATION_ACCESS. It is the only data location permission, and it does not govern reading. AWS is explicit: it "is not needed to query or update underlying data. This permission applies only to creating Data Catalog resources." It answers "may this principal create a table pointing at this prefix", not "may this principal read it".

Three states, not two

The migration becomes tractable once you stop thinking of Lake Formation as a switch. A Data Catalog resource is in one of three states, and the third one is the whole reason the migration is survivable:

State	What decides access	What it costs you to be here
AWS Glue resource	IAM policies and the bucket policy alone. `IAMAllowedPrincipals` holds `Super`, so Lake Formation permissions are not enforced.	Nothing works differently, and nothing is governed. This is where every catalog starts and where a freshly installed Lake Formation still is.
Lake Formation resource	Lake Formation permissions, enforced. The location is registered and `IAMAllowedPrincipals` has been revoked.	Enforcement begins the instant you revoke, for everybody at once. Every job that was relying on IAM alone breaks in the same minute.
Hybrid resource	Both, per principal. Opted-in principals need Lake Formation permissions and IAM permissions; everyone else continues on IAM alone.	A per-principal ramp instead of a cliff. Hybrid access mode enforces only `CREATE_TABLE`, `CREATE_PARTITION` and `UPDATE_TABLE` by default.

Note what hybrid mode does not do: it does not weaken enforcement for the principals you opted in. They need both sets of permissions, which is strictly more restrictive, not less. What it buys is the ability to move one team at a time.

Why This Architecture Holds Up

Tag algebra is the only part that scales

Named-resource grants — "principal P gets SELECT on table T" — are correct and unmaintainable. AWS gives the arithmetic directly: with the named resource method the number of grants required is n(P) x n(R); with tag-based access control using a single LF-Tag, the total of grants to principals and assignments to resources is n(P) + n(R).

At the scale in the challenge above — 200 principals across 900 tables — that is 180,000 grants against 1,100 operations. And 1,100 overstates it, because tables inherit LF-Tags from databases and columns inherit from tables, with inherited values overridable. Tagging 40 databases covers 900 tables; the exceptions are where you spend the assignments.

The number that matters is not the initial write. It is the revoke. Reversing a person's access under the named-resource model means finding every grant they hold across 900 tables; under LF-TBAC it means removing one tag expression from one principal.

The filter lives with the data, not with the engine

The alternative implementations of column and row security are all engine-local: a Redshift RLS policy, an Athena view, a Spark job that projects a subset. Each of them is a correct answer to the requirement and each of them is scoped to one engine, so a lake read by three engines carries three copies of the policy, drifting independently. A Lake Formation data filter is attached to the table in the catalog, and every integrated engine reads the same one.

Filters apply only to read operations, so SELECT is the only permission you can grant with one. That constraint is doing something useful: it means the filter cannot be confused with a write control, and a principal who can write can never be one who sees a partial table.

Cell-level security is an intersection, and intersections compose

A data filter carries a column specification and a row filter expression. All columns plus a row expression is row-level security; specific columns plus the all-rows wildcard is column-level security; both together is cell-level. One primitive, three behaviours, which is why the model stays small enough to reason about.

It composes across accounts too, and in the safe direction. When a row-restricted grant is made to an external account and that account's data lake administrator re-grants it internally, the principal's effective predicate is the intersection of the two. Grant dept='hr' to the account, and a principal separately granted country='us' sees only rows where both hold. A re-grant downstream can narrow access and cannot widen it.

Key Architecture Decisions

1. Register in hybrid access mode, and never flip a live location to enforced

Registering a location with enforcement on is a single action that changes the access rules for everything under that prefix, for everyone, immediately. On a four-year-old lake nobody knows the full list of principals reading it, which means the blast radius is unknown by definition.

Hybrid access mode turns that into a per-principal migration: opt in one team, grant them Lake Formation permissions, confirm their jobs still run, move to the next. AWS does add one caution worth honouring — it does not recommend converting a location that is already registered with enforcement back to hybrid, though it is possible. Go hybrid first, on the way in. It is a ramp, not a rescue.

Two ordering details that will bite otherwise. Before switching an already-registered location to hybrid, opt in the principals currently using Lake Formation permissions and grant Super to IAMAllowedPrincipals on the tables, or existing access breaks in the switch. And for cross-account sharing from a hybrid location, update the cross account version settings to version 4 first — that version carries the RAM policies needed when IAMAllowedPrincipals holds Super.

2. Take away direct S3 access, or you have built a reporting tool

This is the decision the whole post exists for. Registering a location does not remove anyone's ability to read the bucket; it adds a way for engines to read it without their own permissions. Until the bucket policy and the IAM policies stop granting s3:GetObject on the lake prefixes to consuming roles, the Lake Formation grants are advisory.

The end state is that the role registered with Lake Formation is the only identity with read access to the data prefixes, and every consuming principal reaches the data exclusively through an integrated engine. Anything that cannot go through an integrated engine — a bespoke reader, a partner's own tooling, a Lambda parsing Parquet directly — needs an explicit, named, reviewed exception, because it is one.

Check this before you believe any of the rest

Take a consuming role, remove its Lake Formation grants entirely, and run aws s3 cp against a data file as that role. If the bytes come back, the governance model is decoration and every column filter in the catalog is a suggestion.

3. Design the tag vocabulary before granting anything

LF-Tags are a key with a set of allowed values — classification=public,internal,restricted, domain=finance,retail,supply. The vocabulary is a data model, and like any data model it is far cheaper to get right on paper than to re-derive across 900 tagged objects.

The quotas are generous enough that they will not shape the design, which is itself worth knowing so nobody optimises against them: per Region you get 1,000 LF-Tags per account, 1,000 values per LF-Tag, 50 LF-Tag policies per principal per resource type, 10,000 registered paths, a registered path length of 700, and 20 subfolders in an S3 path. All are adjustable.

One that is not generous, and does shape the design: 30 data lake administrators per Region. Data lake administrator is a powerful role — among other things, only a data lake administrator can opt principals into hybrid access mode — and the low ceiling is a hint about how many you are meant to have. Grant it to a small number of automation roles and named people, not to a team.

4. Know the filtering constraints that quietly invalidate a design

These are the ones that turn up after the model is drawn, and each of them has redrawn one:

Constraint	What it forecloses
You cannot apply column filtering to partition keys.	Any design that hides a partitioning dimension — hiding `region` while partitioning by it. Choose partition keys knowing they are permanently visible.
A principal with `SELECT` on a subset of columns cannot also hold `ALTER`, `DROP`, `DELETE` or `INSERT` on that table.	The "writer who also queries a redacted view of their own table" role. Split it into two roles, which is the better design anyway.
Granting `SELECT` with the grant option and column filtering requires an include list, not an exclude list.	Delegated administration expressed as "everything except these two columns". Delegation has to enumerate what is allowed.
Cell-level security is not supported on nested columns, views, or resource links.	Row-plus-column policies on a struct-heavy schema, and on the resource links that cross-account consumers query through. Row and column filtering separately are supported on nested columns; the combination is not.
`array` and `map` are not supported in row filter expressions; `struct` is.	Predicates over a tags map or an events array. The discriminator has to be a scalar or a struct field.
100 data filters for a single principal on a table.	Per-tenant filters granted to one shared role. There is no limit on filters defined per table — the ceiling is per principal, which is exactly the shape a naive multi-tenant design hits.
Nested fields can be included or excluded to five levels.	Deeply nested event schemas. Below five levels, the filter cannot address the field.
Glue 5.0 or higher supports fine-grained access control via Lake Formation only for Hive and Iceberg tables.	Assuming Spark jobs inherit filtering on any format. Table format is now an access-control decision, which is the second time in three posts it has been one.

5. Share across accounts with tags, and expect the invitation step

Cross-account grants are implemented through AWS RAM, and the behaviour differs by relationship in a way that shows up as "the share silently did nothing": if the grantee account is in the same organization, the resource is available immediately; if it is not, RAM sends an invitation that the grantee account's data lake administrator must accept before anything appears. Partner shares fail here, not in the grant.

Two more things to set before the first cross-account share rather than after. Granting to organizations or OUs with LF-TBAC requires cross account version settings at version 3 or above. And Athena and Redshift Spectrum require resource links in the consumer account to include shared resources in queries — the consumer sees the share and still cannot query it until the link exists.

Tradeoffs and alternatives

Prefix-per-tenant with bucket policies

Still the right answer when the boundary genuinely is a prefix and there is one engine. It costs nothing, adds no dependency, and is understood by everyone. It cannot express a column or a row, and it multiplies by the number of engines — each one needs its own policy saying the same thing.

Engine-native row and column security

Redshift RLS and Athena views are mature and often faster to ship for a single consumer. Choose them when the data is read by one engine and will stay that way. The moment a second engine arrives, the policy exists twice and the second copy is the one that will be wrong.

Physical separation into curated copies

Redacted copies per audience are unambiguous and need no runtime authorisation at all. They cost storage, a pipeline per copy, and a permanent question about whether the copies agree. Reasonable for a small number of stable audiences; unworkable as a general model.

What Lake Formation costs

Creating permissions, and their use by integrated services, is at no charge. The costs are indirect: a dependency on the Lake Formation endpoint in the query path, a role with broad read access on the lake, and the operational surface of a second authorisation model that must be reasoned about alongside IAM rather than instead of it.

Closing Thought

The thing worth carrying out of this is that Lake Formation governs a path, not a dataset. It is a permission model that engines opt into by asking for credentials, and its filtering is applied by those engines on the way out. That is a perfectly sound design — it is how the same policy reaches Athena, EMR, Glue and Redshift Spectrum without being written four times — but it means the control is only as strong as the absence of every other path to the same bytes.

Which inverts the usual order of work. The catalog grants are the visible part, and they are the part that produces a satisfying screen full of green. The part that decides whether any of it is true is the S3 policy you removed afterwards, and nothing in the Lake Formation console will tell you whether you removed it. A governed lake is not the one with the most grants. It is the one where the registered role is the only identity that can open the file.

The rest is arithmetic and sequencing. Tag expressions instead of named grants, because n(P) + n(R) is a permission model a team can still revoke from in year three and n(P) x n(R) is not. Hybrid access mode instead of a flip, because the list of principals reading a mature lake is not knowable in advance and a per-principal ramp does not require it to be.

Next in this series

#34 — Streaming versus batch: the latency and cost curve. Six posts have been about the shape of the data at rest and who may read it. The next one asks what it costs to stop waiting — and where on the latency curve the price stops being linear.

Official AWS Reference

Topics: AWS Architecture AWS Architecture Series

AWS Architecture Series #33 — Lake Formation and Governed Access