A namespace is not a boundary

Lab Overview

A developer needs somewhere to run a container. In most organisations that is a ticket: somebody senior creates a space on the cluster, sets limits so the new team cannot consume everything, and wires up permissions so their application reaches its own storage and nobody else's. It takes days, and it is done slightly differently every time.

The problem. Kubernetes looks like it already solved this. It has namespaces, creating one takes a second, and it appears to divide the cluster. It does not. A namespace is a name: it enforces no CPU limit, blocks no network traffic, and grants no cloud permissions. Everything people assume it does has to be added deliberately, as separate objects.

The objective. Build a real tenant boundary for two teams on one cluster, then attack it to find out whether it holds.

The outcome. The boundary held. The first test that said so was meaningless, which turned out to be the more useful finding.

Cloud Services and Concepts

What a boundary actually costs to build: three objects per tenant, none of which arrive with the namespace. A ResourceQuota is the ceiling, a NetworkPolicy is the boundary, and an AWS role is the permission.

Kubernetes allows cross-namespace traffic by default, which is the opposite of what most people assume. Until a NetworkPolicy denies it, one tenant's pod can reach another's.

Architecture and Workflow

1. One EKS cluster hosts two tenants, each in its own namespace.
2. Each namespace carries a ResourceQuota, which caps consumption and makes requests and limits mandatory for every pod in it.
3. A NetworkPolicy denies cross-namespace ingress, which Kubernetes otherwise permits.
4. Tenant pods on EC2 nodes get AWS permissions through Pod Identity.
5. Tenant pods on Fargate get them through IRSA, because Pod Identity cannot work there.
6. Each tenant role reaches exactly one S3 bucket and nothing else.
7. An isolation test runs as part of the build and tries to cross every one of those lines.

Design decisions

Two identity mechanisms, which looks like indecision and is not. AWS recommends Pod Identity over IRSA. AWS also offers Fargate so you never manage a node. Follow both recommendations and you get a cluster that cannot work: the Pod Identity agent is a privileged DaemonSet, and Fargate runs neither DaemonSets nor privileged pods.

And the way it fails is the interesting part. The webhook still injects the environment variables, so the pod looks correctly configured. The credential fetch then times out against an agent that was never there. It presents as an IAM misconfiguration, and you can spend a long time auditing trust policies that were right all along.

The isolation test is part of the build. A boundary nobody has attacked is a boundary nobody has tested.

Prerequisites

Cost starts immediately and there is no free tier. The EKS control plane bills $0.10/hr from creation and a NAT gateway adds $0.045/hr — about $0.166/hr fixed for this build. A cost that starts immediately is safer than one that starts in thirty days, because you design around it.

Know who the cluster admin is before you need to be one. On a CI runner it is not you — which is why access entries are declared in Terraform.

Implementation

A cluster with both an EC2 node group and a Fargate profile, two tenant namespaces with their quotas and policies, two S3 buckets, and the roles that connect them. The lab guide carries the full Terraform.

week-18-eks-self-service/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the modules
|   +-- modules/cluster/    VPC, EKS, node group, Fargate profile, access entries
|   +-- modules/tenant/     namespace, quota, NetworkPolicy, bucket, role
|
+-- tests/                  the isolation probes
+-- scripts/                cleanup, including the OIDC provider sweep

The tenant module is the point: one module instantiated twice produces two boundaries built identically, rather than two boundaries built slightly differently by hand on different days.

Validation and Evidence

Four probes, each deliberately trying to cross a line the design says it cannot.

Security and Cost

Security

The identity mechanism is chosen by where the pod runs, not by preference — Pod Identity on EC2, IRSA on Fargate, because Pod Identity cannot work there.

Each tenant role reaches exactly one bucket: s3:ListBucket on the bucket, GetObject/PutObject on its contents, nothing else anywhere.

The IRSA trust policy pins both :sub and :aud. Without :sub, any service account in the cluster can assume the role — and the omission is silent.

NetworkPolicy denies cross-namespace ingress, because Kubernetes allows it by default.

Control plane audit logging is on. A denied cross-tenant call is only visible there.

Cost

$2.00 measured from Cost Explorer on 16 September — not a calculation. At publication AWS had not posted the charges, so the section originally gave the rate and the duration and left the multiplication to the reader.

After teardown: $0.00, which is what a verified destroy looks like on a bill.

Troubleshooting and Lessons

The test passed against pods that never ran

Problem. The first run reported two passed, two failed. The two failures looked like a real isolation problem.
Root cause. Every probe was rejected before a container started. The ResourceQuota requires every pod to declare CPU and memory requests and limits; my probes declared none, so the admission controller refused all four. The checks looked only for denial strings in the output — and a pod that never starts produces none. So "no denial found" scored as success on the two tests that expected success.
Resolution. Add resources to the probes — and make anything resembling a pod failing to start a hard failure regardless of what the output says.
Lesson. A test that cannot distinguish "denied" from "never ran" is not a test. Two green ticks against rejected pods is worse than a red run, because nobody investigates a pass.

Adding a quota changes the contract for every existing pod

Problem. Pods that previously deployed fine began being refused.
Root cause. A ResourceQuota makes requests and limits mandatory in that namespace — retroactively, for everything.
Resolution. Declare resources on every workload before applying the quota.
Lesson. The quota is not only a ceiling; it is a new admission requirement.

Two AWS recommendations that are mutually exclusive

Problem. Pod Identity silently failed on Fargate.
Root cause. The agent is a privileged DaemonSet; Fargate runs neither. The webhook still injects the environment variables, so the pod looks correctly configured and the credential fetch simply times out.
Resolution. IRSA for Fargate pods, Pod Identity for EC2 pods, in the same cluster — which is AWS's own answer.
Lesson. It presents as an IAM misconfiguration. You can audit correct trust policies for a long time.

Cleanup and Key Takeaways

The script checks clusters, NAT gateways, elastic IPs, VPCs, instances, roles, buckets and log groups by name, then does a tag search on Week=18 for anything the name checks missed.

OIDC providers are on that list deliberately. IRSA creates one per cluster, they do not appear anywhere in the EKS console once the cluster is gone, they survive a careless teardown, and IAM caps an account at 100 of them. They cost nothing, which is exactly why they accumulate.

Key takeaways

• A namespace is a label. The quota is the ceiling, the NetworkPolicy is the boundary, and the AWS identity is the permission. None of them arrive on their own.
• A test that cannot tell "denied" from "never ran" is not a test.
• Two AWS recommendations can be mutually exclusive. Pod Identity and Fargate are each the right default and together produce a silent failure.
• Adding a quota changes the contract for every pod in the namespace, retroactively.
• A cost that starts immediately is safer than one that starts in thirty days, because you design around it.

References

• The full lab guide
• EKS Pod Identity
• IAM roles for service accounts (IRSA)
• Kubernetes NetworkPolicy
• Amazon EKS pricing