GCP Architecture Series #68 — Routes: System-Generated and Custom

Business Challenge

Seven posts have been about where addresses come from. This one is about how a packet leaves. The mechanism is not a device: each VM instance has a controller that is kept informed of all applicable routes from the network's routing table, and each packet leaving a VM is delivered to the appropriate next hop of an applicable route based on a routing order. There is no router to log into; there is a table, and a copy of the relevant parts of it alongside every VM.

The useful way to sort that table is not system-generated versus custom, which is how the documentation titles it. It is by who controls each entry — because that produces three tiers rather than two, and the third tier is invisible. There are routes you create, routes Google creates and maintains on your behalf, and routes that exist in every VPC network, appear in no listing, and cannot be removed. The third group is small, specific, and the reason a locked-down network still answers a health check.

Architecture

Sorted by control rather than by name, the table has three tiers. The evaluation order between them is #69's subject; what matters here is who puts each entry there and who can take it away.

Subnet routes, and the three posts they settle

Each subnet has at least one subnet route for each IP address range that is associated with the subnet — and the lifecycle list reads as a summary of this whole block of the series. A subnet route is created or changed when you add or delete a subnet, when you expand a primary IPv4 range, when you add or delete a secondary range, when you add or delete an IPv6 range, and when Google Cloud adds a new region, which automatically adds a new subnet to auto VPC mode networks.

That last clause is #62 arriving in the routing table. An auto mode network gains a subnet when Google opens a region, which means it gains a route — a destination you did not choose appearing in the table that decides where your packets go. And the uniqueness rule is #63's collision constraint restated at the routing layer: destination ranges for all types of subnet routes must be unique, across local, peering and NCC subnet routes alike. The reason a range you pick today can block a peering next year is that both would produce a subnet route with the same destination.

Subnet routes also inherit the network's global scope from #61: local subnet routes apply to the whole VPC network. Not to a region, not to a zone — every VM in the network knows how to reach every subnet range in it, with no route of yours involved.

Custom routes: the two kinds, and what each obliges

Static routes are the ones you write, and they carry an obligation that is easy to miss because it lives on the next hop rather than on the route. For instance-based routing or an internal passthrough Network Load Balancer as a next hop, you must configure the backend VMs or the next hop instance to forward packets from any source IP address, which means you enable IP forwarding ( can-ip-forward ) on a per-VM basis when you create the VM.

Dynamic routes are the ones BGP writes, and their defining property is that they leave as readily as they arrive: if a next hop for a dynamic route becomes inaccessible, the Cloud Router that manages its BGP session instructs the VPC network to remove the dynamic route. That is the correct behaviour and it means a hybrid path can disappear from your routing table because something failed at the other end of a tunnel. Their scope is not global by default either — routes apply to VMs according to the VPC network's dynamic routing mode. And they are constrained by what is already there: a dynamic route takes destinations that don't conflict with subnet routes or static routes.

One scoping note on policy-based routes, which sit ahead of everything: policy-based routes are evaluated before any other type of route, and policy-based routes are never exchanged through VPC Network Peering. A peer cannot inherit your policy-based routing, which is worth knowing before designing around it in a shared estate.

Why This Architecture Holds Up

This is the part worth reading slowly, because it is both an operational answer and an addressing constraint. Each of these ranges follows the same template in the documentation: the range is used exclusively within Google's production network, and can't be used for any other purpose, Google's edge routers drop internet packets that spoof it, Google’s edge routers also don't have valid next hops for destinations in these ranges, every VPC network carries a non-removable route for it, and you may not use it as a subnet range.

Two consequences follow, and they point in opposite directions. The first is reassurance: a network with no default route and no external addresses still has working SSH through IAP, working health checks, working private DNS forwarding and working load balancer data paths, because every one of those rides a path that does not consult the routes you manage. The documentation makes the load balancer case explicit twice over — forwarding rule paths do not depend on the default route, and backend VMs and target instances don't need to have IP forwarding enabled.

The second is a constraint you cannot negotiate. Five ranges are permanently unavailable for your subnets, and they are not in the prohibited-ranges table from #63 — they are documented here, in prose, in a section about routing. An addressing plan built only from the valid-ranges table can therefore propose a range the platform will refuse, and the refusal will arrive at subnet creation with a reason that sounds unrelated to anything you were reading.

Why this architecture produces that shape

It is worth asking why the invisible tier exists at all rather than appearing as ordinary routes you could see and leave alone. The answer is in the design: there is no router, only a per-VM controller holding the applicable routes. For Google's own control planes to reach your VMs — a health prober, an IAP proxy, a DNS forwarder — those controllers must hold return paths for ranges that are not yours and must not be removable, because removing them would silently disable administration and monitoring of the network while leaving the network apparently healthy.

So the invisibility is not an oversight. It is what makes those functions independent of your configuration, in the same way #66's four reserved addresses are what make a range legible as a subnet. The cost is the same in both cases: something real is happening that the resource listing does not mention.

Key Architecture Decisions

The one to look at today

List the routes in one network and sort them into the three tiers. The custom ones are the inventory you are responsible for; check each still has a reachable next hop and a reason to exist. Then check whether the default route is still present and whether you want it to be — if every VM in that network is meant to egress through Cloud NAT, the default route being there means the ones with external addresses are not. Finally, take the five special-path ranges and grep your addressing plan for them, because that is a conflict the plan will not otherwise reveal until a subnet creation fails.

Closing Thought

The thing I keep running into in this block is that the resource is not the mechanism. A subnet is not a broadcast domain, a gateway address does not answer, an external address is not on the interface, and now a routing table is not the set of paths in effect. In each case the listing is accurate about what it describes and incomplete about what is happening, because the platform implements the behaviour somewhere the resource model does not reach.

What makes routes the clearest example is that the gap is load-bearing in both directions at once. The invisible routes are what let you strip a network down — no default route, no public addresses — and still administer it, which is a genuinely good design. The same invisibility is what makes five address ranges quietly unavailable and puts that fact in a routing document rather than the addressing one. The habit that has paid all week applies again: when something works that has no visible reason to, or fails for no visible reason, the explanation is usually a documented mechanism that the console was never going to show you.

Official Google Cloud Reference

• Routes overview
• Static routes overview
• Subnets
• VPC networks
• Use routes
• Policy-based routes overview
• VPC quotas and limits