Azure Architecture Series #70 — NAT Gateway and SNAT Port Exhaustion

Business Challenge

#68 established that new virtual networks no longer get implicit internet egress, and that you must pick an explicit method instead. Microsoft names a favourite without hedging:

“Azure NAT Gateway is the method that we recommend for outbound connectivity.”

The pitch is that it removes the problem it is named after — it “dynamically allocates source NAT (SNAT) ports to automatically scale outbound connectivity and minimize the risk of SNAT port exhaustion”, and “doesn’t have the same limitations of SNAT port exhaustion as default outbound access and outbound rules of a load balancer.”

Minimise, not eliminate. So the useful question is what the remaining limits are, and the arithmetic is worth doing properly because the headline number and the binding number are different by a factor of twenty.

The fourth row is the post. A million ports is not a pool you draw from freely, because ports are not interchangeable — what a port identifies is a flow, and flows to the same place must differ:

So the inventory is effectively per-destination. Spread across a thousand destinations, a million ports is enormous. Pointed at one — one payment gateway, one API, one database endpoint — the ceiling is 50,000 per public IP, and the way to raise it is to add public IPs rather than to add machines.

Architecture

A subnet-level service with no routes to write, a port inventory shared on demand, and a set of timers that decide how fast ports come back.

Where the ports come from, and who shares them

There is no per-machine allocation, which is the structural difference from load balancer outbound rules: “SNAT port inventory is provided by the public IP addresses, public IP prefixes, or both attached to a NAT gateway. SNAT port inventory is available on demand to all instances within a subnet attached to the NAT gateway. No preallocation of SNAT ports per instance is required.”

That on-demand model is why a NAT gateway tolerates uneven load: one busy VM can draw on the whole inventory rather than being capped at its share. The flip side is that the inventory is a shared resource with a shared fate — “When multiple subnets within a virtual network are attached to the same NAT gateway resource, the SNAT port inventory… is shared across all subnets.” One noisy subnet can starve another.

One mitigation that costs nothing to know: “TCP and UDP are separate SNAT port inventories”, so a UDP-heavy workload cannot exhaust TCP capacity or vice versa.

The timers, which are the real exhaustion mechanism

Ports are not returned the moment a connection ends. Two kinds of delay apply, and together they set the recycling rate:

The idle timeout is the one worth not touching, and the documentation says so explicitly: “When a connection goes idle, the NAT gateway holds onto the SNAT port until the connection idle times out. Because long idle timeout timers can unnecessarily increase the likelihood of SNAT port exhaustion, we don’t recommend that you increase the TCP idle timeout duration to longer than the default time of 4 minutes.”

Which inverts the instinct. Faced with connections dropping, the obvious move is to raise the idle timeout — and that makes exhaustion more likely, because every idle connection is now squatting on a port for up to two hours. The documented alternative is keepalives: “TCP keepalives appear as duplicate acknowledgements (ACKs) to the endpoints, are low overhead, and are invisible to the application layer” — they refresh the flow without holding a port idle.

UDP keepalives come with a catch worth reading twice: “Unlike TCP connections, a UDP keepalive enabled on one side of the connection applies only to traffic flow in one direction. You must enable UDP keepalives on both sides.”

And a caveat on the whole table, which is unusually candid: “These timer settings are subject to change. The provided values can help with troubleshooting. You shouldn’t take a dependency on specific timers at this time.” Useful for diagnosis, not for capacity planning.

The precedence ladder

Attaching a NAT gateway changes outbound behaviour for things you did not touch, because it outranks them: “Azure NAT Gateway takes precedence over other outbound connectivity methods, including a load balancer, instance-level public IP addresses, and Azure Firewall.”

The full order, highest first:

Two consequences. A machine with its own public IP stops using it for egress the moment a NAT gateway appears on the subnet — and the change is not retroactive, since “There are no drops in traffic flow for existing connections that use other explicit methods”, so old flows persist on the old path while new ones move. And a 0.0.0.0/0 UDR to a firewall beats the NAT gateway entirely, which is how a hub-and-spoke design keeps inspection in place.

Why This Architecture Holds Up

Because the number that constrains you is not the number that gets quoted

“Over a million SNAT ports” is true and nearly useless for capacity planning. The figure to size against is “up to 50,000 concurrent connections per public IP address to the same destination endpoint”.

So the question is not how many connections the workload opens, but how many it opens to each place. A fleet making a million connections spread over thousands of endpoints is comfortable. A fleet making 60,000 simultaneous calls to one API endpoint is over the limit on a single public IP, regardless of how much inventory is sitting unused.

Which makes the remedy counterintuitive and cheap: attach more public IP addresses. Each adds 64,512 ports and another 50,000-connection allowance to that destination. Scaling out the VMs does nothing for this limit; scaling out the addresses is the whole fix.

The second ceiling is absolute and worth knowing because its failure mode is ugly: “Each can support up to 2 million active connections simultaneously… If a NAT gateway exceeds 2 million connections, the data path’s availability declines and new connections fail.” Counted on the 5-tuple, so a connection is a flow and not a socket.

Because the fix for dropped connections makes exhaustion worse

This is the trap I would most expect an on-call engineer to walk into. Connections are dying; the idle timeout is right there and goes up to two hours; raising it looks like the fix. It is the opposite of the fix, because the port stays held for the whole idle window.

And the symptom that sends people there is itself informative. A NAT gateway “sends a TCP Reset (RST) packet for any connection flow that it doesn’t recognize as an existing connection… When the sender… receives the… TCP RST packet, the connection is no longer usable.” So an application that has been idle past the timeout does not hang — it gets a reset on its next write, which reads like the remote end hung up.

There is one small mercy in the timer table: a reset frees its port after 16 seconds versus 65 for a graceful FIN. Connections that die badly recycle four times faster than connections that close politely, which is a genuinely strange property to hold in mind.

Because it silently takes over egress from things you configured

The precedence ladder means adding a NAT gateway is not an additive change. A VM with an instance-level public IP keeps the address — and stops using it for outbound. A load balancer with outbound rules keeps the rules — and stops being the egress path. Azure Firewall likewise, unless a UDR puts it above.

For most estates that is what you want, and it is why the migration is simple. But it makes the NAT gateway a thing that changes the meaning of existing configuration without changing the configuration — the same shape of problem as AllowVNetInBound in #65, arriving from the opposite direction. Anyone auditing egress addresses needs to read the ladder, not the resource list.

Because the SKU choice is permanent and drags the public IPs with it

Two SKUs, and the difference is resilience: Standard is “zonal (deployed to a single availability zone)” while StandardV2 is “zone redundant” with higher throughput, IPv6 and flow logs. The consequence of choosing Standard is explicit — “If the zone that’s associated with a Standard NAT gateway goes down, the outage affects outbound connectivity for the subnets associated with the NAT gateway.”

And there is no path between them: “You can’t upgrade the Standard SKU to the StandardV2 SKU. You must create a StandardV2 NAT gateway to replace the Standard NAT gateway on your subnet.” Nor can you carry the addresses across — “The StandardV2 SKU requires StandardV2 public IP addresses or prefixes. Standard public IPs aren’t supported with StandardV2.”

Which lands directly on #69: the Standard-v2 public IP that could only be used with a Standard v2 NAT Gateway turns out to be the matching half of this decision. Choosing zone redundancy here means re-IPing, and re-IPing means every firewall rule and A record that referenced the old addresses. The zone cannot be changed later either — “After you deploy a NAT gateway, you can’t change the zone selection.”

Because the Basic SKU exclusion is subnet-wide

A constraint that turns #69's migration into a prerequisite rather than a parallel task: “You can’t use Azure NAT Gateway with subnets that have Basic resources. Resources for the Basic SKU, such as a Basic load balancer or Basic public IPs, don’t work with Azure NAT Gateway.”

So one leftover Basic public IP anywhere in the subnet blocks the recommended outbound method for everything in it. Given #68 pushes new subnets towards requiring an explicit method and #69 established that Basic is retired and unsupported, the three posts compose into a single ordering: clear the Basic resources, then attach the NAT gateway, then make the subnet private.

Because a handful of exclusions will decide the design before the numbers do

The fragmentation one is the quiet repeat. #68 noted that default outbound access IPs “don’t support fragmented packets”, and the recommended replacement does not either. A workload failing on large UDP payloads will fail identically after the migration, which is worth knowing before anyone proposes the migration as the cure.

Because live migration breaks flows and nothing warns you

One operational detail with no configuration to fix it: “NAT Gateway doesn’t support connection continuity for existing flows when a virtual machine undergoes live migration. Existing connections through the NAT gateway might be disrupted and need to be reestablished.”

Live migration is a platform action you do not schedule. So long-lived outbound connections through a NAT gateway need application-level reconnection logic, not because of anything in your design, but because the host underneath can move. That belongs in a resilience review rather than a network one.

Key Architecture Decisions

Closing Thought

The honest summary is that a NAT gateway makes SNAT port exhaustion much harder to hit and changes what hitting it looks like. The old failure was a per-instance allocation running dry while the pool had capacity. The new failure is a per-destination ceiling: 50,000 concurrent connections from one public IP to one endpoint, with a million ports sitting idle because they are not the kind of port you need.

That is why the capacity question is about shape rather than volume. Tell me how many connections your fleet opens and I cannot size this. Tell me how many it opens to its busiest single endpoint and the answer is arithmetic: divide by 50,000, round up, attach that many public IP addresses. Everything else — instance count, subnet count, VM size — is irrelevant to that number.

And the thing I would put on the wall is the idle timeout inversion. Connections are resetting, the timeout is adjustable up to two hours, and raising it is exactly wrong because every idle connection holds its port for the whole window. Microsoft says so plainly and the control is still there, set to four minutes, inviting the mistake. Keepalives refresh a flow without holding a port; a longer timeout holds a port without refreshing anything. One of those scales and the other is a slow way to run out.

Official Azure Reference

• NAT gateway resource — the 64,512 ports per address, the per-destination SNAT semantics, the 50,000 and 2 million connection ceilings, the full timer tables, the bandwidth splits and the limitations list
• What is Azure NAT Gateway? — the recommendation, the precedence ladder over other outbound methods, the two SKUs and the Basic-resource exclusion
• Source Network Address Translation with Azure NAT Gateway — further reading on the allocation mechanics (no claims drawn from it)
• Troubleshoot Azure NAT Gateway — the connectivity validation steps referenced for ICMP (no claims drawn from it)