Business Challenge
The previous post sorted routes by who controls them and deferred the question of which one wins. The documentation answers it carefully: there might be more than one applicable route for a given packet, and the following steps model the process that Google Cloud uses to select a route. The word doing the work is steps. This is not a comparison that ranks candidates on one scale; it is a sequence of eliminations, and several of the stages do not eliminate so much as terminate.
That structure explains three things that look like inconsistencies when you expect a priority contest. Longest-prefix matching, the rule every network engineer reaches for first, applies at exactly one stage and only to two of the route types. Priority means something different depending on where in the sequence you are. And three of the stages are settled by an algorithm Google does not publish — with the same documented remedy each time.
Only in one place. The most-specific-destination step is explicit about what it is looking at: Google Cloud determines which applicable static or dynamic routes have the most specific destination that contains the destination IP address of the packet. Static or dynamic. By the time that step runs, subnet routes and policy-based routes have already been resolved or discarded, and for policy-based routes specificity never applied — Google Cloud evaluates policy-based routes solely by their priority, and the selected policy-based route might not be the most specific match for the packet's source or destination.
Correct approachWork out which stage your routes are competing at before reasoning about which wins. Two static routes to overlapping destinations: specificity decides. A policy-based route against a static route: the policy-based route is evaluated first and specificity is irrelevant. A subnet route against anything: see the next card.
You cannot. For most subnets, if a packet's destination matches the destination range of a regular subnet route, Google Cloud exclusively uses the subnet route, and then all other routes are disregarded, and evaluation stops at this step. No custom route of any specificity or priority is consulted. And the terminal case is a drop rather than a fallthrough: if the packet's destination isn't associated with a resource or belongs to a stopped VM instance, the packet is dropped.
Correct approachIntra-subnet steering is not a routing problem on this platform — it needs a policy-based route, which is evaluated before subnet routes, or a design that puts the traffic on a different destination range. And note what the drop means for diagnosis: traffic to an address inside one of your subnets that has no live resource behind it disappears silently. It does not escape to a default route, because the subnet route already won.
It is not promised. ECMP is hash-based — Google Cloud uses a five-tuple hash if the packet contains port information; otherwise, it uses a three-tuple hash — but the documentation then withdraws the guarantee you would infer from that: consistent next hop selection for a given hash is not guaranteed, and Google Cloud might direct packets to a different next hop even if the hash is the same and the route model is unchanged.
Correct approachNever put a stateful device behind ECMP across multiple next hops. If a NAT instance, firewall VM or proxy holds per-connection state, two equal routes to two such instances will break connections, and intermittently rather than reproducibly. Use a single next hop, or an internal passthrough Network Load Balancer, which exists to hold that affinity — and which the model explicitly never load-balances between: Google Cloud doesn't perform ECMP among multiple internal passthrough Network Load Balancer next hops.
Architecture
The model is worth reading as a pipeline with exits, because the exits are the part that surprises. Four stages can end evaluation before any custom route is considered.
| Stage | How it decides | Can it end evaluation? |
|---|---|---|
| Special routing paths | If a special routing path is applicable, your route selection model contains only the special path | Yes, absolutely |
| Policy-based routes | Priority alone; ties go to an internal algorithm | Yes — a match disregards all non-policy-based routes |
| Subnet routes | Matching at all. Specificity never enters it | Yes, exclusively — and drops if nothing live is behind the address |
| Private Service Connect endpoints for global Google APIs | Destination match — each endpoint is an internal forwarding rule that acts as a route in the VPC network | Yes — the packet goes to a Google Front End |
| Most specific destination | Longest prefix, static and dynamic routes only | No — it narrows to identical destinations |
| Most favourable custom route type | Local custom routes are preferred over NCC dynamic routes, and NCC dynamic routes are preferred over peering custom routes | No |
| Unusable next hops | Eliminates next hops that are invalid, stopped, deleted or not established | No |
| Priority | Discards all routes except for those with the highest priority | No |
| Preference category | Four bands; Google Cloud doesn't perform equal-cost multipath (ECMP) among routes that belong to different preference categories | No |
| Send or drop | One route sends; two or more means ECMP; none means ICMP unreachable | Terminal by definition |
What the exits mean in practice
The first exit is the invisible tier from #68 asserting itself: a special path, if applicable, is the only thing in the model. You cannot override IAP or health check return paths with a route of your own, which is the design's whole point.
The policy-based exit is the one worth designing around, because it is the only documented way to pre-empt a subnet route. When a policy-based route matches and is not configured to skip other policy-based routes, Google Cloud delivers the packet to the next hop internal passthrough Network Load Balancer, and disregards all non-policy-based routes. That is how you insert an inspection appliance into traffic that would otherwise go straight to its subnet neighbour — which answers the gap #61 opened, where there was no choke point between subnets.
And the subnet exit is the hard floor. Two posts established that subnet routes are automatic, global, and must have unique destinations; this one adds that they are also unconditional. The only route type that pre-empts them is policy-based.
Where the platform picks for you
Three stages are resolved by an unpublished algorithm, and each carries its own warning. For policy-based routes: if your route selection model includes two or more matching highest-priority policy-based routes, Google Cloud selects a single policy-based route by using an internal algorithm, and the advice is direct — to avoid this ambiguity, we recommend that you create policy-based routes that have unique priorities.
For peering: Google Cloud uses an internal algorithm to import peering custom routes from a single VPC network, and the selection is not even stable over time — the peer network that Google Cloud selects might change if your VPC network peers with a new VPC network or if it disconnects from an existing peer VPC network. Adding an unrelated peering can therefore move traffic that was working.
For Network Connectivity Center the pattern repeats — Google Cloud uses an internal algorithm to select hybrid spokes from a single routing VPC network — with the same fix: ensure that NCC dynamic routes have unique priorities in each hybrid spoke if hybrid spokes come from two or more different routing VPC networks. Across all three, priority is not expressing a preference between paths you are happy with either way. It is the mechanism by which you decline to let the platform choose. Equal priorities are a decision to accept a non-deterministic outcome.
Why This Architecture Holds Up
Before anything is selected, the model removes next hops that cannot work — and the list is more specific than "is it up".
| Next hop condition | Documented behaviour |
|---|---|
| Wrong kind of address | The next-hop-address of a static route must match an IP address that is assigned to a VM in the route's VPC network. If it matches a different type of resource (like an alias IP range) or doesn't match any resource, Google Cloud disregards the route |
| Next hop VM stopped or deleted | Google Cloud disregards each static route whose next hop VM instance has been stopped or deleted |
| Classic VPN tunnel not established | Google Cloud disregards each static route with a next hop Classic VPN tunnel that doesn't have an active Phase 1 (IKE) security association (SA) |
| Guest software dead | Google Cloud doesn't validate whether the guest OS of a next hop VM or a backend VM for a next hop load balancer is processing packets |
The first row is #64 reappearing: an alias IP range is not eligible as a static route next hop, so the service addresses you so carefully separated cannot be pointed at. The last row is the important one for operations. The platform checks that a next hop exists and is running; it does not check that the thing inside it is working. A next hop VM whose forwarding daemon has crashed stays in the route model, keeps attracting packets, and nothing in the routing layer will notice. Health checking that is a job for a load balancer, not for a route — which is one reason the internal passthrough Network Load Balancer next hop exists.
Preference categories, and why ECMP stops at their edges
The penultimate stage sorts the survivors into four bands and keeps only the best, because Google Cloud doesn't perform equal-cost multipath (ECMP) among routes that belong to different preference categories. In order: one or more static routes with next hop instances or Classic VPN tunnels; then one or more dynamic routes of a single type; then a single static route with next hop internal passthrough Network Load Balancer; and last, static routes with next hop default-internet-gateway.
Two consequences follow that are easy to get wrong. A static route with an instance next hop and a dynamic route to the same destination at the same priority do not share traffic — the static route wins the band and the dynamic route is discarded, so a hybrid path you believed was an active peer is not carrying anything. And the internet gateway sitting in the least-preferred band is what makes a more specific or differently-typed route reliably pre-empt general egress.
A packet with no route left behaves visibly: if your route model is empty, Google Cloud drops the packet with an ICMP type 3, code 0 (network unreachable) message. A packet that matched a subnet route with nothing live behind the address is dropped with no such message. So the presence or absence of an ICMP unreachable is a real diagnostic: unreachable means nothing matched, and silence means something matched and had nowhere to deliver. One more silent case worth knowing: if a next hop load balancer doesn't have global access enabled, Google Cloud drops all packets sent from VM instances, VLAN attachments, and Cloud VPN tunnels in regions different from the load balancer's region.
Key Architecture Decisions
| Situation | Decision | Because |
|---|---|---|
| Any two routes that could both match | Give them distinct priorities | Three stages fall back to an internal algorithm on ties, and one of those can change when peerings change. |
| Inserting inspection between workloads in one subnet | A policy-based route | It is the only type evaluated before subnet routes, which are otherwise unconditional. |
| A stateful next hop — NAT, firewall, proxy | One next hop, or an internal passthrough load balancer | ECMP does not guarantee a flow stays on one path, and never spans load balancer next hops. |
| Expecting a hybrid path to share load with a local static route | Check the preference bands first | Instance and Classic VPN next hops outrank dynamic routes; the dynamic route is discarded, not blended. |
| Pointing a static route at a service address | Use a primary internal address, not an alias range | An alias IP range is the wrong resource type and the route is disregarded. |
| Relying on a VM next hop | Monitor the forwarding software yourself | The platform does not validate whether the guest OS is processing packets. |
| Load balancer next hop used across regions | Enable global access | Otherwise packets from other regions are dropped silently. |
The one to look at today
Find every set of routes in one network that share a destination, and check whether any pair shares a priority. Each such pair is a place the platform will choose for you, and for peering routes the choice can move the next time somebody adds an unrelated peering. Then look at anything acting as a stateful next hop and count how many equal routes point at it — more than one is a connection-resetting bug waiting for enough traffic to become visible.
Closing Thought
What makes this model hard is not its length. It is that the vocabulary is borrowed from a system that worked differently. On a router, longest prefix is the rule and priority breaks ties within it. Here, specificity is one stage among many, priority is a different mechanism at three separate points, and the two concepts never meet for most route types. Reading the documented steps once is less work than the alternative, which is inferring a model from behaviour that is consistent but not the one you assumed.
The sentence I will keep from this is the ECMP one, because it is unusually candid: consistent next hop selection for a given hash is not guaranteed, even with the same hash and an unchanged route model. Most platforms let you believe hash-based balancing is deterministic and leave you to find out otherwise. Stating it plainly turns an intermittent production mystery into a design constraint you can respect in advance. Nine posts into this block, that has been the pattern worth trusting: the documentation is more precise than the interface, and the precision is almost always load-bearing.
#70 moves from choosing a path to permitting traffic along it: VPC firewall rules — the default rules, the implied ones, and what a rule is actually attached to.
Comments