Business Challenge
Every hybrid product in the last few posts required one of these. NCC's hybrid spokes wrap them; HA VPN and every flavour of Interconnect mandate them. So it is worth being precise about what the thing is, because the name is misleading in a way that matters.
Cloud Router is a distributed and fully managed offering that lets you connect your on-premises networks through Border Gateway Protocol (BGP) speaker and responder capabilities. And then the sentence that should reset the mental model: Cloud Routers don't provide packet routing or forwarding capability. The division of labour is explicit — Andromeda handles all packet routing and forwarding capabilities, and Cloud Routers manage corresponding BGP sessions.
It is a BGP speaker whose output is routes, not a device packets pass through. Three consequences follow, and the third is the one I did not expect.
There is nothing to size. It forwards no packets, so it has no bandwidth, no throughput figure and no datapath. Its job is to create dynamic routes in VPC networks based on BGP-received and custom learned routes, working alongside Cloud Interconnect, Cloud VPN and Router appliances. The capacity that matters is a control-plane one — how many unique prefixes it may hold — not a data-plane one.
Correct approachPlan Cloud Router against route counts and quotas, and plan throughput against the attachment or tunnel underneath it. A useful tell that the product is control-plane only: Cloud NAT relies on Cloud Routers for control plane capabilities, but not for BGP session management — a product that uses a Cloud Router while speaking no BGP at all.
Surviving is the problem. A static route stays installed when the path beneath it dies, which turns a link failure into a blackhole. The alternative is a custom learned route, and the documentation states the difference as the selling point: unlike static routes , custom learned dynamic routes are automatically withdrawn when the BGP session for a next hop goes down.
Correct approachPrefer a custom learned route wherever the next hop is a hybrid attachment. Custom learned routes are manually configured on Cloud Router to simulate routes learned from a remote peer, and they exist precisely for the case where you cannot make the far end advertise properly: custom learned routes are useful when you don't have administrator control to configure a remote peer router. For some next hops there is no other option — custom learned routes are the only way to define non-BGP routes for certain next hops, like Cloud Interconnect VLAN attachments and HA VPN tunnels.
Not in the default configuration. Legacy is the default best path selection mode when a new VPC network is created, and in it, AS path information is only relevant within a single Cloud Router BGP task, and the dynamic route control plane doesn't use the AS path information to make best path selection decisions. Google says it outright: don't rely on selecting the best path based on AS-path length information when different Cloud Router software tasks are involved for the same destination prefix.
Correct approachIf you need AS path to work the way BGP says it should, you need standard mode, where AS path information is considered over all of the routes learned on all of the Cloud Routers of the VPC network and this lets you influence traffic flow using AS path prepending. But read the next section before switching, because Google recommends against it by default.
Architecture
A BGP speaker, two control planes behind it, and two independent choices — how paths are selected and what gets advertised.
Two control planes, and why the distinction shows up in behaviour
Routes do not go straight from BGP into the network. Both types of Cloud Router learned routes are processed by a dynamic route control plane and then a VPC network control plane. That two-stage shape explains several things that otherwise look arbitrary: quotas are enforced at the first stage, AS path is evaluated differently at each stage in legacy mode, and the hand-off between them is where routes get dropped.
Feature-wise the BGP implementation is conventional and reasonably complete: BGP session management, including support for Bidirectional Forwarding Detection (BFD) and MD5 authentication, and BGP route policies let you set rules to filter BGP routes or modify BGP route attributes. Worth noting against #82, which found BGP communities unsupported in NCC: route policies are the mechanism you get instead.
One behaviour is important for anyone drawing transit through Google: Cloud Router doesn't re-advertise learned routes unless you use an NCC hybrid spoke with data transfer enabled. So by default this is not a transit router. It learns, it installs, and it stops — which is the same non-transitive instinct #81 found in peering, expressed in the hybrid layer.
The best path modes, and why the recommendation is the odd one
Cloud Router offers two modes for best path selection: standard mode and legacy mode. Legacy is the default, and the guidance is unusually conservative: we recommend that you use legacy mode unless you need specific features of standard mode or use a Google Cloud feature that relies on standard mode.
That is a strange sentence once you read what each mode does. Standard mode is the standards-compliant one — the standard best path selection mode more closely adheres to RFC 4271 — and it is the one in which AS path behaves as a network engineer expects. Legacy is the one in which it does not. Google is recommending the less correct mode as the default, and I think the reason is visible in the NCC interaction below: legacy is predictable across VPC boundaries, and standard is not.
| Legacy (default, recommended) | Standard | |
|---|---|---|
| AS path scope | Within a single BGP task only; the dynamic route control plane ignores it | Considered over all of the routes learned on all of the Cloud Routers of the VPC network |
| AS path prepending | Unreliable across tasks | Works — lets you influence traffic flow using AS path prepending |
| Standards adherence | — | More closely adheres to RFC 4271 |
| MED comparison | Smallest MED, after AS path and origin type | Configurable: always compare, or group by neighbour AS, via routingConfig.bgpAlwaysCompareMed |
| Same prefix from several routing VPCs (NCC) | Legacy selection applied consistently across them | The best route from one of the routing VPCs based upon an internal algorithm |
The last row is the decisive one, and it is the fourth appearance of this pattern in the block. In standard mode, where the same prefix arrives from several routing VPCs, the workload VPCs use the best route from one of the routing VPCs based upon an internal algorithm. Google's guidance follows directly: if you plan on advertising the same route from multiple routing VPCs, then we recommend that you select legacy best path selection for all your VPC networks.
Read the two modes as a trade between local correctness and global determinism. Standard mode gives you RFC-conformant selection within a VPC network and the ability to steer with AS path — genuinely better, for one network. Across several routing VPCs on an NCC hub it degrades to an internal algorithm you cannot inspect. Legacy mode never promised AS path across tasks, so it has nothing to degrade: the same inputs give the same answer everywhere. The lesson is the one #78, #81 and #82 kept circling — where this platform cannot make a choice deterministic, it picks one quietly, and the defensive move is to avoid constructing the ambiguity rather than to reason about how it resolves.
The tie-break ladder, and the one global rule above it
Inside a single BGP task the selection is standard BGP, in three steps. First, shortest AS path length : the BGP task sorts the list of next hops by AS path length, from shortest AS path length to longest AS path length. Then origin: the BGP task prefers next hops that use the Interior Gateway Protocol (IGP) over next hops that use the Exterior Gateway Protocol (EGP), and the BGP task prefers next hops that use EGP over next hops that have an Incomplete origin type. Then smallest MED value : The BGP task sorts next hops from smallest to largest multi-exit discriminator (MED) value.
MED is the attribute that actually does the work here, because it survives the hand-off to the dynamic route control plane where AS path does not. Two details make it usable. Custom learned routes participate: the priority of a custom learned route is treated like a MED value, so a hand-written route competes on the same scale as a learned one. And in global dynamic routing mode an inter-regional cost is added to the MED of routes arriving from other regions, which is how region affinity gets expressed — the mechanism #84 covers properly.
Above all of that sits one rule that overrides the entire exercise, and it connects straight back to #69 and #70: dynamic routes always have a lower VPC network route priority compared to static routes. No amount of MED or AS path tuning makes a dynamic route win against a static route for the same destination. If a hybrid path is being ignored, look for a static route before debugging BGP.
Why This Architecture Holds Up
Advertisement is configured in two places, and the interaction is the part worth knowing before someone changes one of them.
Advertised routes are Cloud Router BGP prefixes advertised to its BGP peers, and the advertisement mode is configurable on both the Cloud Router level and BGP session level. Each level offers default or custom, where Cloud Router custom advertisement mode gives you complete control over IP address advertisements.
The precedence is not symmetric. A session left on default inherits whatever the router is set to, default or custom alike. But a session set to custom detaches completely: the BGP session doesn't inherit the advertisement configuration of the Cloud Router. So one session on a router can advertise an entirely different set of prefixes from its siblings, and changing the router-level configuration afterwards will not touch it.
A team adds a prefix at the router level to fix reachability, verifies it on one session, and does not notice that a second session — set to custom months earlier, perhaps for a single exception — silently ignored the change. The symptom is asymmetric reachability over redundant links, which reads as a peer problem rather than a configuration one. Auditing advertisement therefore means reading every session's mode, not the router's.
Three smaller rules round out what actually leaves the router. Subnet advertisement tracks reality by itself — Cloud Router automatically updates subnet route advertisements whenever a subnet lifecycle activity occurs — so new subnets appear to peers without a configuration change, which is convenient and worth knowing if a peer's prefix limit is tight.
Custom and subnet advertisement can collide, and the collision is silent: a custom IP address range is ignored if it exactly matches a subnet route that a Cloud Router advertises according to the dynamic routing mode of its VPC network. So a custom entry duplicating a subnet does nothing — and if the dynamic routing mode later changes which subnets are advertised, the custom entry's effect changes with it.
And IPv6 has a hole worth naming: subnet route advertisement never includes advertisement of external subnet IPv6 address ranges. They can only be advertised as custom ranges. Combined with the overview's exclusions — IPv6 BGP peering and IPv6 route exchange aren't supported for the following resources: Classic VPN tunnels Router appliance (part of Network Connectivity Center) Cross-Cloud Interconnect VLAN attachments — an IPv6 hybrid design needs checking against the specific attachment type before it is drawn.
One more warning is easy to skip and expensive to discover: if you advertise privately used public IPv4 addresses to another network, that other network might not be able to access internet resources at those public IPv4 addresses. You are telling a peer that a public range is reachable through you, and it will believe you for all traffic to that range, including traffic meant for the real internet host.
The quota that deletes destinations
This is the operational edge, and it has the same shape as #82's hub quota. Over the limit, the behaviour is not a rejected change: the dynamic route control plane drops all route information associated with unique destinations that sort beyond the limit for either Cloud Router quota. Whole destinations go, not individual paths.
Two mitigations are documented and both are worth wiring up. There is an audit trail — you can identify which routes were dropped by viewing route_quota_events in Cloud Logging — which makes this one of the few failures in this block with a named log to query rather than an inference to make. And there is an honest statement about what survives the hand-off: the dynamic route control plane only guarantees that it sends the routes with the lowest MED values to the VPC control plane. Routes with higher MEDs may or may not arrive, so a backup path's presence in the VPC routing table is not guaranteed by its being learned.
Put that last sentence next to the earlier one about dynamic routes ranking below static routes and the practical rule emerges: the VPC routing table is a filtered, truncated projection of what BGP learned, and the filtering happens in two places with different rules. Debugging a hybrid path means checking what BGP received, then what the dynamic route control plane kept, then what the VPC control plane installed — three questions, not one.
Key Architecture Decisions
| Situation | Decision | Because |
|---|---|---|
| Capacity planning a hybrid link | Size the attachment or tunnel, not the Cloud Router | Cloud Routers provide no packet routing or forwarding capability. |
| Pinning a route over a hybrid next hop | Custom learned route, not a static route | It is withdrawn automatically when the BGP session drops. |
| A peer that will not advertise correctly | Custom learned routes | They exist for when you lack administrator control of the remote router. |
| Wanting AS path prepending to work | Standard best path selection | Legacy mode's control plane ignores AS path entirely. |
| Advertising one prefix from several routing VPCs | Legacy mode on all of them | Standard mode resolves that case by an internal algorithm. |
| Steering traffic reliably | Use MED, and set custom learned route priority deliberately | MED survives the hand-off to the dynamic route control plane; AS path may not. |
| A hybrid route that is learned but not used | Look for a static route first | Dynamic routes always rank below static routes. |
| Auditing what you advertise | Read every BGP session's mode, not the router's | A session set to custom does not inherit the router's configuration. |
| A custom range that appears to do nothing | Check whether it exactly matches an advertised subnet | An exact match is ignored. |
| Advertising external IPv6 subnet ranges | Add them as custom ranges | Subnet advertisement never includes them. |
| IPv6 over Classic VPN, Router appliance or Cross-Cloud Interconnect | Do not plan on it | IPv6 BGP peering is unsupported on those three. |
| Advertising privately used public IPv4 ranges | Expect the peer to lose internet access to those addresses | It will route them to you instead. |
| Operating near the route quota | Alert on route_quota_events |
Over quota, whole destinations are dropped rather than changes refused. |
| Relying on a learned backup path | Verify it in the VPC routing table | Only lowest-MED routes are guaranteed to reach the VPC control plane. |
The one to look at today
Check the best path selection mode on each VPC network that has a Cloud Router, and check whether anybody is using AS path prepending to steer traffic. If the mode is legacy — which it will be, unless somebody changed it — that prepending is not doing what its author believes across Cloud Router tasks. Then list the advertisement mode of every BGP session rather than every router, because a session set to custom is invisible to router-level changes and is where asymmetric reachability comes from.
Closing Thought
The useful correction in this post is the smallest one: Cloud Router forwards nothing. Once that lands, a lot of confusion dissolves — why there is no throughput figure, why it exists per-region rather than per-link, why Cloud NAT can use one without speaking BGP, and why its quotas are measured in prefixes. The name borrows the authority of a physical device and describes a different kind of object: a managed BGP speaker whose only output is entries in somebody else's routing table.
The finding I will keep is the recommendation to use legacy mode. It is the only place in this series where Google has advised against the more standards-compliant option, and the reason turns out to be the same determinism problem the whole block has been circling. Standard mode is better within one VPC network and ambiguous across several; legacy is worse within one and consistent across all of them. Twenty-three posts in, the recurring lesson is not about any one feature — it is that this platform's hardest trade-offs are between local correctness and global predictability, and it resolves them by quietly preferring predictability, in defaults whose names make them sound like the thing you are supposed to grow out of.
#84 takes the setting this post deferred to three times: dynamic routing mode, regional versus global — which decides how far a learned route travels and what it costs in MED to get there.
Comments