The cache that answered a question it was never asked

Lab Overview

A building has a locked door, and a guard who checks your badge against the room number before letting you through. It works. Then someone installs a faster side entrance, because the queue at the door was annoying. The side entrance has its own guard, and that guard was told only "staff may enter."

Nobody removed the badge check. It just stopped being on the route people take.

The problem. Multi-tenant systems pool tenants into one DynamoDB table and keep them apart with an IAM condition: dynamodb:LeadingKeys, which restricts a caller to its own partition key. The control is real. But it is evaluated by DynamoDB, on the request DynamoDB receives — and a cache in front of the table changes who receives the request.

The objective. Build the pooled table and the condition, prove the isolation holds, then put a DAX cluster in front for latency and make the identical call again.

The outcome. tenant-alpha asking for tenant-beta's item is refused with AccessDeniedException going direct, and returns tenant-beta's data through DAX. Same function, same role, same table, same item. Two more documented traps in the same policy behaved the same way. Ten checks, ten matches.

Cloud Services and Concepts

Pool or silo, and why anyone pools

You can give every tenant its own table — a silo — and isolation becomes a resource boundary rather than a policy. The limit is arithmetic: AWS allows 2,500 tables per account per Region, raised to a maximum of 10,000 on request, after which its own guidance is more accounts. A pooled table has no such ceiling, and pays for it by making isolation a property of a policy rather than of the furniture.

A condition that passes when nobody asks

Two of this week's three findings come from one sentence in AWS's documentation: "A ForAllValues condition evaluates to true when the condition key is not present in the request." A Scan carries no partition key. So a policy that grants Scan alongside a LeadingKeys condition reads as though it restricts the caller to one tenant, and returns every tenant. AWS's fix is a companion Null condition requiring the key to be present at all.

Architecture and Workflow

Three Lambda functions run byte-identical code. What differs is the policy on each execution role:

• alpha and beta carry the policy AWS documents — LeadingKeys scoped to their own tenant, a Null companion, Select pinned to SPECIFIC_ATTRIBUTES, and no Scan.
• audit carries the same policy with both guards removed and Scan added. It is not a straw man: it is what you get by writing the obvious policy and testing it with GetItem.

All three also hold dax:* read permissions on the cluster, because that is what a team adds when read latency becomes a problem. That grant is the whole story.

Prerequisites

Implementation

What was deployed

Thirty-two resources in all, applied in about twelve minutes, nearly all of it the DAX cluster.

The two policies, side by side

Everything separating the guarded policy from the one that leaks is these two condition blocks:

"Condition": {
  "ForAllValues:StringEquals": {
    "dynamodb:LeadingKeys": ["tenant-alpha"],
    "dynamodb:Attributes":  ["tenant_id","item_id","description","amount"]
  },
  "Null":                 { "dynamodb:LeadingKeys": "false" },
  "StringEqualsIfExists": { "dynamodb:Select": "SPECIFIC_ATTRIBUTES" }
}

Remove the last two lines and add dynamodb:Scan to the actions, and you have the audit role.

Validation and Evidence

Ten calls across three principals and two paths. Each case declares what it expects before it runs, and the script exits non-zero if any case disagrees — three of them expect to be allowed something they should not be, and a script that merely printed what happened would read the same whether the finding reproduced or quietly stopped reproducing.

secret_note is an attribute no policy's allow-list contains. Its appearance in a response is the finding showing itself, rather than a claim about what a policy says.

Security and Cost

What this means for a platform team

The sentence worth taking away: if tenant isolation rests on an IAM condition, putting a cache in front of the table is a security change, not a performance change. It will arrive as a latency ticket, be reviewed by whoever owns performance, and will not mention tenancy.

AWS says so plainly — users "inherit the permissions of the DAX cluster's IAM policy when they access that cluster… No other permissions are recognized", and the use of DAX "can subvert those policies." The documented mitigation is one cluster per isolation boundary, each with its own scoped policy, which prices isolation by the node-hour.

Two smaller controls, both worth copying: add a Null condition beside every LeadingKeys condition, and pin dynamodb:Select to SPECIFIC_ATTRIBUTES whenever you restrict attributes — otherwise omitting a projection returns everything.

Cost

The cluster existed for about fifty minutes. DAX bills per node-hour or partial hour, so fifty minutes is three node-hours, and no deploy-test-destroy cycle of this lab can cost less however fast you are.

Measured: $0.1202. Cost Explorer for 8 November reports $0.1200 against DynamoDB Accelerator and $0.0002 against DynamoDB — the five seeded items and a few hundred reads. The estimate above and the bill agree here, which is worth saying only because they often do not: a rate card tells you the price of a thing, not how many of them a lab turns out to use.

Left running, this is the first week in a while with a meter rather than a bill of cents: $2.88 a day, idle or not.

Troubleshooting and Lessons

Problem. The apply failed with SubnetNotAllowedFault: Subnet … is not in a supported availability zone.
Root cause. This account has a Local Zone enabled (us-east-1-dfw-1a, Dallas), and aws_availability_zones returns names sorted alphabetically. us-east-1-dfw-1a sorts before us-east-1a, because - precedes a. Taking "the first three AZs" put a Local Zone at index 0.
Resolution. Filter the data source on zone-type = availability-zone, which excludes Local Zones and Wavelength Zones.
Lesson. "The first three availability zones" is a sorting assumption wearing the costume of a requirement.

Problem. Terraform replaced all three Lambda functions on every apply, with no code change.
Root cause. The package hashed differently each build. pip writes console-script wrappers into bin/; built on Windows those are .exe files that embed a path, and each dist-info/RECORD stores a hash of them.
Resolution. Delete bin/, the RECORD files and the __pycache__ left by the import check, and write the zip with fixed member timestamps. The package fell from 4.9 MB to 652 KB as a side effect.
Lesson. A Windows executable had been riding inside a package destined for a Linux runtime, and the only symptom was a needlessly churning plan.

Problem. The teardown script reported success without checking anything.
Root cause. It queried a DAX cluster named week26-tenant-isolation. DAX caps cluster_name at 20 characters, so the cluster is actually week26-isolation. The query asked about a name that never existed and reported "gone" regardless — on the one resource billing by the hour.
Resolution. Query the real name. The false pass was proved first, by running the old query against a cluster that was running at the time and watching it report not-found.
Lesson. A safety check that cannot fail is worse than no check, because it is load-bearing in your confidence and nowhere else.

Cleanup and Key Takeaways

Teardown was confirmed against the API rather than the destroy output — a destroy reports what Terraform removed, not what is left in the account. The DAX cluster, the subnet group, the table, the three functions and the VPC were all checked by name, and Terraform's state was confirmed empty.

Next week: week 27.