π In This Post
Lab Overview
Your orders app calls your rates app. How do you control who is allowed to make that call?
The problem. Almost everyone answers with network rules: a security group, a CIDR range, "anything in this subnet can reach port 8080 over there". That controls addresses, not identities. Anything that lands in that subnet can make the call, and the rule cannot tell your orders app from a compromised test box beside it. You also cannot write the sentence you actually mean β "only the orders service may call rates" is not expressible in a security group.
The objective. Build one service that only a named IAM role may call, then try to call it as the wrong one. Two Lambda functions, same VPC, same subnet, same security group, byte-identical code. The only difference is which role signs the request.
The outcome. The control works β and a single optional Terraform argument silently disables it while the console still shows the policy attached and the service healthy.
| Difficulty | Intermediate |
| Lab runtime | Ran 41 hours before teardown (40 billed service-hours) |
| Measured cost | $1.00 for 40 service-hours |
| Deployment | Terraform via HCP Terraform with OIDC |
| Full lab guide | GitHub β prerequisites, Terraform, commands, validation, cleanup |
Not App Mesh. App Mesh shut down on 30 September 2026 and cannot be created. VPC Lattice is where AWS points instead, and that is the only reason App Mesh appears here at all.
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| Amazon VPC Lattice | A managed switchboard between your own applications. Each VPC attaches to a shared service network and each app registers as a service, replacing a web of peering connections and security-group rules nobody can audit. | An application-layer network that connects services to each other and authenticates every call, instead of relying on subnet and security group reachability. |
| Service network | The switchboard itself β a named meeting point that VPCs attach to and services register with. Attaching is free and removes the need for peering. It is also the first place an access policy can be applied. | The Lattice object services join in order to reach one another. It authenticates callers, but on its own it does not decide which caller is allowed to do what. |
| Lattice service | One callable application on that switchboard. Not your code β the object that fronts it, with a stable DNS name resolvable only inside an attached VPC. | A single application registered on the service network, with its own listener, routes and targets. |
| Auth policy | An IAM policy document attached to a service or service network naming which identities may call it. Evaluated by AWS before the request reaches your code. | A resource policy attached to a Lattice service network or service that says which signed principals may call it. Without one, authentication proves who called but nothing refuses them. |
| AWS Lambda | The two callers and the backend. Identical code; only the execution role differs. | Runs a piece of code on demand with no server to manage. You are billed per request and per millisecond it runs, and each in-flight request occupies one execution environment. |
| HCP Terraform | Runs Terraform remotely and assumes an AWS role by OIDC, so no static keys exist anywhere. | HashiCorp's hosted Terraform. It runs plans and applies on its own machines and keeps the state file, and it assumes an AWS role by OIDC so no static keys exist anywhere. |
Two concepts decide the whole week, and they are easy to conflate.
auth_type is a switch set independently on the service and on
the service network. AWS_IAM means evaluate the policy; NONE means
skip that layer entirely. The policy and the switch are different objects and they can
disagree.
SigV4 signing is how a caller proves who it is β the same signing the
AWS CLI performs on every command. A request without a signature is not rejected for being
malformed; it is treated as coming from anonymous, which a policy can allow or
refuse like any other principal.
Architecture and Workflow
- Two Lambda functions run identical code in the same subnet behind the same
security group. One assumes the role
orders-client, the otherstranger. - Each signs its request with SigV4 for the service
vpc-lattice-svcsand calls the service's Lattice DNS name. - The request reaches the service network, whose auth policy allows any non-anonymous caller from that VPC. Both callers pass this layer.
- The request then reaches the rates service, whose auth policy names the
orders-clientrole and theGETmethod. orders-clientgets 200;strangergets 403 β decided by AWS, before either request reaches any code.- Allowed requests pass through a listener on port 80 to a Lambda target group and the backend function, which performs no authorisation of its own.
Design decisions
A Lambda target group, not an ALB. Lattice can forward to IPs, instances, an ALB or a Lambda function. Choosing Lambda means the week runs with no load balancer and no servers β you pay for neither idle compute nor an idle ALB.
Two layers of policy, not one. The service network expresses "everything in this organisation"; the service expresses "this role, this method". Separating them is what makes the failure in Troubleshooting both possible and instructive.
No authorisation in the application. Deliberate β the point is that the check happens in AWS, so the backend cannot forget to implement it.
Prerequisites
| Requirement | Detail |
|---|---|
| Terraform | >= 1.10 |
| AWS provider | >= 6.68 β auth_type is optional on both Lattice resources, which matters below |
| HCP Terraform | A workspace with OIDC to AWS. No static keys |
| AWS permissions | Create VPC Lattice, Lambda, VPC, IAM and CloudWatch Logs resources |
| Region | us-east-1 |
One region assumption will bite you. If your account has opted into a Local
Zone, data.aws_availability_zones returns it alongside the standard zones β
and Lambda cannot attach an ENI in a Local Zone. Filter on zone type rather than taking the
list as given.
No prerequisite labs. Week 23 is self-contained.
Implementation
Twenty-four resources: a VPC with two subnets, a service network and its VPC association, a Lattice service with listener and Lambda target group, two auth policies, three Lambda functions, and the IAM roles that separate them. The lab guide carries the full Terraform and exact commands.
week-23-service-networking-lattice/
|
+-- terraform/
| +-- environments/dev/ HCP backend, OIDC, wires the modules
| +-- modules/network/ VPC and subnets, zone-type filtered
| +-- modules/lattice/ service network, service, listener, both auth policies
| +-- modules/compute/ the three Lambda functions and their roles
|
+-- src/ byte-identical caller code, two different roles
|
+-- scripts/ the six signed test calls
The two callers are 2 of the 24 resources and differ in exactly one attribute β their execution role. Same code, same subnet, same security group. That is what makes the test meaningful: anything that distinguishes them has to be identity.
The control itself is two arguments and a document:
auth_type = "AWS_IAM" # on the service AND the service network
# plus an aws_vpclattice_auth_policy naming the role and method
Both are optional in the provider, and the auth policy is a separate resource that does not
set auth_type itself. Omit it and you get the default, NONE.
Validation and Evidence
Deployment succeeding proves nothing about authorisation. Six signed calls were made against the running service.
| Test | Expected | Actual | Status |
|---|---|---|---|
| Infrastructure deploys | 24 resources | 24 created, apply clean | Pass |
orders-client calls GET | 200 | 200 | Pass |
stranger calls GET | 403 | 403, "no service-based policy allows" | Pass |
| Unsigned (anonymous) call | 403 | 403, "no network-based policy allows" | Pass |
Policy deleted, auth_type still AWS_IAM | Fails closed | 403 β fails closed as documented | Pass |
auth_type set to NONE, policy left attached | Policy still enforced | 200 for the wrong role | Fail β and that is the finding |
AWS IAM,
the policy Active, and inside it one role and one method. The service
network carries a second policy of its own, and each object has its own
authType deciding whether that policy is read at all.
Security and Cost
Security
Identity, not address. The control this lab exists to demonstrate is that "only role X may call method Y" is expressible and enforced by AWS before the request reaches any code. A security group cannot express it.
Two layers, evaluated independently. Both must allow a request. The denial messages name which layer refused β "no network-based policy allows" versus "no service-based policy allows" β which is a genuinely useful diagnostic.
No static credentials. HCP Terraform assumes its AWS role by OIDC.
The accepted trade-off: an auth policy controls who may call, not what the call contains. It is not a WAF and does not inspect payloads.
Not production-ready as built. A real deployment needs the auth policy under
the same review as any other IAM policy, and auth_type asserted explicitly on every
Lattice resource rather than left to the provider default β see Troubleshooting for why
that is not a style preference.
Cost
$1.00 measured, verified against Cost Explorer by usage type: 21 service-hours
on 15 October and 19 on the 16th, at $0.025 each, under
USE1-VPCLattice-Service-Hourly. Data processing came to
$0.0000.
That shape is the thing worth taking away. A Lattice service bills for being reachable, not for being called β an idle service network costs exactly what a busy one does. The only variable that moves the number is how long it is left standing, which is why the week ends with a destroy run rather than a quiet weekend.
A resource gateway is billed at $0.10 per resource-hour, four times the hourly rate of a service, which is why this build does not use one.
Troubleshooting and Lessons
A policy that is attached, visible, and inert
Problem. With auth_type set to NONE, the wrong
role using the wrong method received a 200.
Root cause. auth_type and the auth policy are separate objects.
The policy stayed attached and visible, reporting state: Inactive β a field
almost nobody queries β while the service reported Status: Active at the top
of the same page.
Resolution. Set auth_type = "AWS_IAM" explicitly on both the
service and the service network. Terraform makes it optional on both, and the auth policy
resource does not set it, so omitting it silently yields the NONE default.
Lesson. An attached policy is not an enforced one. Query the state field, not the document.
None, the policy reads Inactive, and the console greys the document out rather than removing it. The service still reports Status: Active at the top of the page.And the system still looked healthy
Problem. With the service policy inert, spot-checking from outside the VPC
still returned 403, suggesting the controls were fine.
Root cause. The service network policy carried on refusing anonymous callers
even though the service layer was disabled. Something was still saying no.
Resolution. Test from inside the VPC with a real signed identity, not from
outside with none.
Lesson. A half-disabled two-layer control looks healthier than a fully
disabled one, because part of it still refuses. That is worse than an obvious failure,
not better.
Three that cost an evening
An opted-in Local Zone. data.aws_availability_zones returned
us-east-1-dfw-1a, a subnet landed there, and Lambda refused to attach an ENI.
Filtering on zone type fixes it portably.
The VPC association security group needs inbound rules. The client makes an outbound call, so an egress-only group looks obviously right. It produces no error at all β requests simply time out, which sends you hunting for a routing problem.
Lattice requires x-amz-content-sha256: UNSIGNED-PAYLOAD. Omit
it and you get IncompleteSignatureException; supply a real hash and you get
"Signed payloads are not supported". Both arrive as 403s from a service whose entire
job is issuing 403s β a bad costume for a signing bug to wear.
Every one of these was found by calling the service, not by reading the configuration.
Cleanup and Key Takeaways
A VPC cannot be deleted while a service network association survives, so a half-finished
destroy leaves a VPC that looks orphaned and is not. The teardown was run through HCP β
run-F6yePbXRy6hMQK3m, 32 resources β so it has an id someone can look up
later.
"Applied" is Terraform's account of itself. The verification puts the same questions to AWS: service networks, services, both association types, target groups, the VPC, the Lambdas, the log groups and the IAM roles.
Nothing is shared and nothing survives. Left running, the only meter is the service itself at $0.025 per hour.
Key takeaways
- An auth policy lets you write the sentence you mean. "Role
orders-clientmayGETthis service" is not expressible in a security group, and it is enforced before the request reaches your code. auth_typeand the policy are different objects that can disagree. A policy can be attached, visible and inert. Assertauth_typeexplicitly β the provider default isNONE.- A half-disabled layered control looks healthier than a broken one. Part of it still refuses, which is exactly what makes it dangerous.
- Lattice bills for reachability, not traffic. An idle service costs what a busy one does; only elapsed hours move the number.
- Signing failures wear the same costume as authorisation failures. Both are 403. Read the message, not the status code.
References
- The full lab guide β prerequisites, Terraform, commands, validation, cleanup
- VPC Lattice auth policies
- Signing requests to VPC Lattice
- VPC Lattice pricing
- Migrating from App Mesh
- Security groups for VPC Lattice
Comments