A namespace is not a boundary

Why — The Problem This Solves

A developer needs somewhere to run a container. In most organisations that is a ticket: somebody senior creates a space on the cluster, sets limits so the new team cannot consume everything, and wires up permissions so their application can reach its own storage and nobody else's.

It takes days, and it is done slightly differently every time. The alternative — handing over the cluster and trusting everyone — is how one team's pod ends up able to read another team's data.

What a boundary actually costs to build

Three objects per tenant, none of which arrive with the namespace:

What You Need to Know — Skills & Tools

Architecture — How It Fits Together

One cluster, two tenants, and deliberately two different ways of granting AWS permissions.

Why two identity mechanisms, which looks like indecision and is not

AWS recommends Pod Identity over IRSA. AWS also offers Fargate so you never manage a node. Follow both pieces of advice and you get a cluster that cannot work.

The Pod Identity agent is a privileged DaemonSet. Fargate runs neither DaemonSets nor privileged pods. So on Fargate, Pod Identity simply does not function — and the way it fails is the interesting part:

AWS's own answer is to run both: Pod Identity for pods on EC2, IRSA for pods on Fargate, in the same cluster. This build does exactly that, one tenant each, so the difference is visible rather than described.

How We Built It — Step by Step

Step 1 — Decide what the tenant boundary is made of

The tenant module is the week. Given a name, it creates a namespace, a ResourceQuota, a NetworkPolicy denying cross-namespace ingress, a private encrypted bucket, an IAM role scoped to that one bucket, and a service account wired to the role.

The identity half switches on where the pods run:

module "tenant_a" {          module "tenant_b" {
  compute       = "ec2"        compute       = "fargate"
  identity_mode = "pod_identity"   identity_mode = "irsa"
}                            }

Same module, same boundary, two routes to it.

Step 2 — The trust policy is where the mechanisms actually differ

Pod Identity trusts one EKS service principal and carries the cluster, namespace and service account as session tags. The role works in another cluster with no edit.

IRSA trusts this cluster's OIDC provider and matches the service account by a string inside the token:

condition {
  test     = "StringEquals"
  variable = "${oidc_provider_url}:sub"
  values   = ["system:serviceaccount:tenant-b:app"]
}

Omit that :sub condition and any service account in the cluster can assume the role. Omit :aud and the token audience goes unchecked. Neither omission produces an error, and neither is visible until somebody tries.

That, in one block, is why Pod Identity exists: reusing an IRSA role in a second cluster means editing this document, the trust policy is capped at 2048 bytes so it holds about four of these, and IAM allows 100 OIDC providers per account.

Pod Identity has no equivalent block. The mapping lives in the EKS API instead, and nothing inside the cluster points at an IAM ARN:

Step 3 — Deploy it

HCP Terraform, VCS-driven, as every week in this series. The run list is a fair summary of how it went:

Verifying it actually works

Four checks, and the two that matter are the ones expected to fail:

1. tenant-a reads its own bucket — must succeed
2. tenant-a reads tenant-b's bucket — must be denied
3. tenant-b reads its own bucket — must succeed
4. tenant-b reads tenant-a's bucket — must be denied

Run 1 and 3 alone and you have shown that permissions work. Only 2 and 4 show that they stop anything.

That is the second run. The first one is the reason this section exists.

Challenges — What Actually Went Wrong

1. The test passed against pods that never ran

The first run reported two passed, two failed. The two failures looked like a real isolation problem, so I went to read the raw output of a single probe rather than reason about it:

The ResourceQuota requires every pod to declare CPU and memory requests and limits. My probes declared none, so the admission controller refused all four. The checks looked only for denial strings in the output — and a pod that never starts produces none. So "no denial found" scored as success on the two tests that expected success.

The quota rule is worth knowing independently: once a namespace has a quota covering CPU or memory, every pod in it must declare both requests and limits. A cluster that worked yesterday starts rejecting deployments the moment someone adds a quota.

2. Locked out of a cluster I had just created

The apply reported success. kubectl said "the server has asked for the client to provide credentials."

bootstrap_cluster_creator_admin_permissions grants cluster admin to whichever identity created the cluster. Terraform ran on a CI runner, so the admin was the runner's role. Nobody else had access — including the account owner, sitting at a laptop.

Access entries are the current mechanism, and they are worth reaching for early. They replaced editing the aws-auth ConfigMap by hand, which was for years the standard way to lock yourself out of a cluster with no way back in.

3. The apply chose a Local Zone

The very first apply failed twice in one run:

UnsupportedAvailabilityZoneException: EKS does not support creating control
plane instances in us-east-1-dfw-1a

NotAvailableInZone: Nat Gateway is not available in this availability zone

aws_availability_zones returns Local Zones alongside standard ones. us-east-1-dfw-1a is Dallas. EKS will not put a control plane there and NAT gateways do not exist there at all.

Neither error message contains the words "Local Zone", and both arrive at apply time — after the VPC has already been built. The fix is a filter most examples omit:

filter {
  name   = "zone-type"
  values = ["availability-zone"]
}

4. A run that reported no changes, because it never saw them

After pushing the access-entry fix I triggered a run through the API, and it returned add=0 change=0. The code was right; the run was using a configuration version from before the push, because the VCS webhook had not ingested the commit yet.

Worse, the webhook's own run was already queued and holding the workspace lock, so my API run sat in pending behind it. The answer was to apply the VCS run and discard the duplicate — but the failure mode to remember is that a plan showing no changes may mean your code is fine and the runner is looking at an older commit.

Security — Controls at Every Layer

• The identity mechanism is chosen by where the pod runs, not by preference — Pod Identity on EC2, IRSA on Fargate, because Pod Identity cannot work there
• Each tenant role reaches exactly one bucket — s3:ListBucket on the bucket, GetObject/PutObject on its contents, and nothing else anywhere
• The IRSA trust policy pins both :sub and :aud — without :sub any service account in the cluster can assume the role, and the omission is silent
• NetworkPolicy denies cross-namespace ingress — Kubernetes allows it by default, which is the opposite of what most people assume
• Control plane audit logging is on — a denied cross-tenant call is only visible there
• Cluster access through access entries, not a hand-edited ConfigMap
• The isolation test is part of the build — a boundary nobody has attacked is a boundary nobody has tested

Cost

Prices as of September 2026 — verify at the EKS pricing page.

This bill is the opposite shape to the last several weeks

Weeks 11, 12 and 16 of this series, and a QuickSight subscription that caught me last month, all cost nothing on the day they were created and began billing thirty days later when a free trial quietly ended. Nothing alerted, because nothing had changed — a number that was zero simply stopped being zero.

EKS is the reverse. It charges about $4 a day from the moment the cluster exists, visibly, with no trial and nothing deferred. That is easier to manage, not harder, and it changes the discipline: teardown becomes part of the build rather than an afterthought. This cluster was created and destroyed inside twelve hours by design.

Cleanup

./scripts/cleanup.sh

It checks clusters, NAT gateways, elastic IPs, VPCs, instances, roles, buckets and log groups by name, then does a tag search on Week=18 for anything the name checks missed.

OIDC providers are on that list deliberately. IRSA creates one per cluster, they do not appear anywhere in the EKS console once the cluster is gone, they survive a careless teardown, and IAM caps an account at 100 of them. They cost nothing, which is exactly why they accumulate.

References

• Pod Identity and IRSA compared — AWS's own table of the limits that drove the change
• EKS Pod Identity restrictions — including Fargate
• Fargate on EKS — what you give up
• Kubernetes versions in standard support
• EKS pricing
• This week's code

Key Takeaways

• A namespace is a label. The quota is the ceiling, the NetworkPolicy is the boundary, and the AWS identity is the permission. None of them arrive on their own
• A test that cannot tell "denied" from "never ran" is not a test. Two green ticks against pods rejected before startup is worse than a red run, because nobody investigates a pass
• Two AWS recommendations can be mutually exclusive. Pod Identity and Fargate are each the right default, and together they produce a silent failure
• Adding a quota changes the contract for every pod in the namespace — requests and limits become mandatory, retroactively
• Know who the cluster admin is before you need to be one. On a CI runner it is not you
• A cost that starts immediately is safer than one that starts in thirty days, because you design around it

What I'd do differently in production

This is a same-day lab cluster with two tenants and one operator. Several decisions above are only correct because of that.

• A NAT gateway per availability zone. One NAT is a single-AZ failure away from breaking egress for every pod in the other subnet. It doubles a $0.045/hr line to remove a single point of failure — obviously right in production, and wasteful for twelve hours
• Narrow the API endpoint. public_access_cidrs is open here. In a real estate it is your office and your CI ranges, or the endpoint is private and reached through a bastion or VPN
• Generate the tenant, do not hand-write it. The module is the mechanism; self-service needs a front door — a pull request template or a ticket that fills in the variables — so a new team never edits Terraform directly
• Add LimitRange alongside ResourceQuota. The quota makes requests and limits mandatory; a LimitRange supplies sensible defaults so developers are not forced to guess on every pod
• Egress policy too. The NetworkPolicy here denies cross-namespace ingress. A compromised pod's first move is usually outbound, and that is a separate rule
• Do not build on Ingress NGINX. Upstream Kubernetes retired it in March 2026 — no further bug fixes or security patches. Gateway API or the AWS Load Balancer Controller instead