Business Challenge
A platform team runs about 300 EC2 instances and a few dozen ECS services across three Availability Zones, all in private subnets. Egress goes through NAT gateways, which is the textbook layout. When Finance asked what the third-largest line on the AWS bill was, the answer was NAT gateway data processing.
That number is worth pausing on, because a NAT gateway is not a service anyone deliberately buys capacity from. It is plumbing. The team went looking for what was generating the traffic, expecting a runaway job. What they found instead was ordinary: container images from ECR, logs to CloudWatch, Systems Manager agent traffic, Secrets Manager lookups, S3 reads. Almost none of it was going to the internet at all. It was AWS talking to AWS, routed out through a NAT gateway and billed $0.045 per GB for the privilege.
A NAT gateway bills two ways: $0.045 per hour for existing, and $0.045 per GB for every gigabyte it processes. The per-GB charge does not care whether the destination is a public website or an S3 bucket in the same Region. A private subnet with a default route to a NAT gateway sends all AWS API traffic through it, and pays processing on every byte.
FixTreat the default route as a fallback, not a highway. Anything with a private path to the service should be taking it.
AWS is unambiguous: there are no data processing or hourly charges for using gateway endpoints. They are free. They are also available for exactly two services — Amazon S3 and DynamoDB — and they work by adding a prefix-list route to your route tables rather than by using PrivateLink at all.
This is the single highest-return change available, and it is skipped constantly, because nothing breaks without it. Traffic still works. It just costs money.
FixAdd gateway endpoints for S3 and DynamoDB to every VPC, in every account, as a baseline. There is no cost case to make.
For everything else — ECR, CloudWatch Logs, SSM, Secrets Manager, STS, KMS — the private path is an interface endpoint, and those are billed. You pay for each hour the endpoint remains provisioned in each Availability Zone, plus data processing at $0.01 per GB for the first petabyte each month, falling to $0.006 and then $0.004 per GB at higher volumes.
The per-AZ part is what surprises people. One endpoint across three AZs is billed as three. Ten services across three AZs is thirty hourly charges, in every VPC that needs them. A change made to save money can comfortably fail to.
FixCount endpoints as services × AZs × VPCs before committing. Below a traffic threshold, NAT processing is genuinely cheaper.
This is the constraint that reshapes hybrid designs, and AWS states it plainly. In its own comparison table, gateway endpoints “do not allow access from on premises” and “do not allow access from another AWS Region”. Interface endpoints allow both.
The mechanism explains the rule. A gateway endpoint is a route-table entry pointing at an AWS-managed prefix list, and route tables only govern traffic originating inside the VPC. Traffic arriving over Direct Connect, a VPN, or a Transit Gateway attachment never consults them. It is also why traffic to the same service in a different Region goes to the internet gateway instead: prefix lists are Region-specific.
FixA hybrid estate needs an interface endpoint as well as the gateway endpoint, not instead of it. See the next section for why that pairing is the point.
Architecture
The order to make the decisions in
The three paths are not alternatives to choose between once. They are a sequence, and the order matters because each step removes traffic from the step below it.
First, gateway endpoints for S3 and DynamoDB. Free, no data processing, no hourly charge. For most workloads this is the largest single block of traffic and the easiest to move. There is no threshold analysis to do because there is no cost.
Second, interface endpoints for the chatty control-plane services. ECR pulls are large and constant. CloudWatch Logs is continuous. SSM, Secrets Manager and STS are small per call but relentless. These are where the per-GB comparison actually favours an endpoint: $0.01 per GB against NAT's $0.045, before the hourly charges on either side.
Third, keep a NAT gateway for what is genuinely internet-bound. OS patching, package registries, third-party APIs, webhooks. The goal was never to delete the NAT gateway. It was to stop routing AWS traffic through it.
The pairing that makes hybrid work
If on-premises systems also need S3, the naive reading of the constraint above is that the free gateway endpoint is useless and everything must move to a billed interface endpoint. That is not the case, and the option that prevents it is easy to miss.
When you enable private DNS on an S3 interface endpoint, AWS offers Enable private DNS only for inbound endpoints — and it is selected by default. With it on, in-VPC applications keep resolving S3 to the gateway endpoint, which is free, while on-premises applications reaching the VPC through an inbound Resolver endpoint use the interface endpoint, which is billed. Each class of traffic takes the cheapest path that can carry it, with no client changes.
AWS enforces the dependency: the setting requires a gateway endpoint to exist in the VPC. Set
it without one and the API returns
To set PrivateDnsOnlyForInboundResolverEndpoint to true, the VPC must have a gateway
endpoint for the service. You also cannot delete the gateway endpoint while the option
is selected — clear the option first.
What interface endpoints for S3 will not do
Before standardising on them, note the documented gaps. Interface endpoints for S3 do not
support FIPS endpoints, website endpoints, legacy global endpoints, or
CopyObject and UploadPartCopy between buckets in different Regions.
On the transport side they do not support TLS 1.0, TLS 1.1, TLS 1.3, or
hybrid post-quantum TLS. That TLS 1.3 exclusion sits oddly in the middle of the list and is
worth checking against your own client configuration before you find it in an incident.
Why This Architecture Holds Up
The comparison is per-GB against per-AZ-hour
An interface endpoint is not automatically cheaper than a NAT gateway, and pretending otherwise is how these projects disappoint. The endpoint replaces a $0.045 per-GB charge with a $0.01 per-GB charge plus a fixed hourly charge in every AZ it lives in. The saving is $0.035 per GB, and it has to cover those hourly charges before it is a saving at all.
That makes traffic volume the deciding variable, per service. A service pulling terabytes through NAT pays for its endpoint many times over. A service making a few thousand small API calls a month may never break even, and is better left on the NAT path. The instinct to “endpoint everything” produces a longer bill, not a shorter one.
Partial hours round up, everywhere
AWS bills each partial NAT gateway-hour as a full hour. The same provisioned-hour model applies to interface endpoints per AZ. Neither is metered by the second, so short-lived environments — ephemeral test VPCs, per-branch preview stacks, anything torn down and rebuilt through the day — pay disproportionately. In those environments the right answer is often a single shared NAT gateway and no endpoints, which inverts the production recommendation entirely.
Routing is longest-prefix, so the endpoint quietly wins
A gateway endpoint adds a prefix-list route, and AWS resolves conflicts by longest prefix
match. With a 0.0.0.0/0 route to an internet gateway or NAT gateway in the same
table, the endpoint route is more specific and takes precedence for that service in that
Region. You do not need to remove the default route, and nothing needs reconfiguring at the
application layer.
The corollary is that this is invisible. Adding the endpoint changes the bill and not the behaviour, so there is no signal that it worked and no signal when someone removes it. Verify with cost data or VPC flow logs, not by whether the application still functions — it will either way.
Key Architecture Decisions
| Decision | Choose this | Because |
|---|---|---|
| S3 and DynamoDB from inside the VPC | Gateway endpoint, always | No hourly charge and no data processing charge. There is no volume at which this loses. |
| S3 from on-premises | Interface endpoint, plus keep the gateway endpoint | Gateway endpoints do not allow access from on premises. Private DNS for inbound endpoints only keeps in-VPC traffic free. |
| High-volume services (ECR, CloudWatch Logs) | Interface endpoint | $0.01 per GB against $0.045 per GB clears the per-AZ hourly charge quickly at scale. |
| Low-volume services | Leave on the NAT path | Per-AZ hourly charges apply whether the endpoint carries one gigabyte or none. |
| Ephemeral or short-lived VPCs | Shared NAT gateway, minimal endpoints | Both models bill provisioned hours and round partial hours up, which punishes short lifetimes. |
| Genuine internet egress | Keep the NAT gateway | Endpoints reach AWS services only. Patching and third-party APIs still need a route out. |
The audit that pays for itself
Enable VPC flow logs on the private subnets and group accepted traffic by destination prefix list. The output ranks, in bytes, exactly which AWS services are currently paying NAT processing. That ranking is the endpoint backlog, ordered by return, and it usually shows two or three services carrying most of the cost. Do those, measure, and stop — the tail is where endpoint charges start exceeding the traffic they displace.
Closing Thought
Nobody designs a NAT gateway bill. It accumulates from a default route that was correct when the VPC had one workload in it, and it grows quietly because nothing ever breaks. The traffic flows, the applications work, and the only symptom is a line item that looks like infrastructure rather than a decision.
The fix is not a migration. It is two gateway endpoints that cost nothing, a short list of interface endpoints chosen on measured volume rather than instinct, and a NAT gateway kept deliberately for the traffic that actually needs the internet. What makes it worth doing carefully is that the wrong version — endpoints everywhere, in every AZ, in every VPC — produces a bill that is larger than the one you set out to reduce, and it is just as invisible.
Comments