Home› Blog› Week 13 — HA VPN and BGP: The Route Nobody Wrote
GCP Weekly Lab GCP Terraform

The route nobody wrote

Two VPCs, two VPN tunnels, and a route that appears in no Terraform file — because the routers told each other about it. Then break one tunnel and time how long the other takes to notice.

Verified against current vendor documentation on 15 November 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.
GCP Platform Engineering Lab · Week 13 of 58

Lab Overview

The problem. Building an encrypted tunnel between two networks is the easy part. Both ends agree on ciphers, the tunnel comes up, and a dashboard turns green. Then a packet leaves a machine on one side headed for an address on the other, and the network has to answer a question nobody has answered yet: does this go down the tunnel, or out to the internet? Something has to put that answer in a route table. Type it by hand and it is correct until the day the other side changes — or until the tunnel dies, and the route keeps pointing at it with total confidence.

The objective. Build HA VPN between two VPCs, let BGP populate both route tables, then break one tunnel and measure how long the network takes to route around it.

Expected outcome. Two route tables neither side typed, each holding the other's subnet twice — once per tunnel. Kill one tunnel and the count halves, traffic keeps flowing, and no human touches anything. If the prefix only appears once, the second tunnel is not carrying it and the redundancy is nominal.

Deployment method. Terraform on HCP Terraform agents with no stored credentials — the workspace mints an OIDC token per run and exchanges it for a short-lived Google token, with separate identities for plan and apply. Built in Week 02; every week since rides on it.

Cloud Services and Concepts

ServicePurpose hereWhat it is
HA VPNCarries the traffic between the two VPCsGoogle Cloud's managed IPsec VPN. A gateway has two interfaces with their own public IPs; using both is what the availability SLA is defined on.
Cloud RouterRuns BGP and programs what it learns into the VPCA managed BGP speaker. It holds your ASN and decides what the route table says — it is not in the data path, so traffic never flows through it.
BGPFills both route tables, and empties them when a tunnel diesThe protocol networks use to tell each other which address ranges they can reach. The same one that runs the public internet between ISPs.
VPC and subnetsThe two networks being joinedA VPC is a private network; a subnet is a range inside it tied to one region. Custom mode means no subnet exists until you declare it.
Compute EngineTwo probe VMs, one per sideVirtual machines. Here they are instruments rather than the subject — something has to exist at each end for a route to lead to.
Terraform & HCP TerraformDeclares all of it and applies it remotelyTerraform describes infrastructure as code; HCP runs it on hosted agents, authenticating per run through Workload Identity Federation so no key is stored.

Static routes versus BGP, and why it is not a style choice

A route table says: for addresses in this range, send the packet that way. For your own subnets it fills itself in. For anything beyond a tunnel, something has to add the entry.

You can type it — anything for 10.30.0.0/24, send down tunnel 1. Or the two routers can tell each other, continuously, which ranges they can reach. That conversation is BGP, and each side identifies itself with an ASN; there is a private range, 64512–65534, for cases like this.

The difference that matters is not convenience. BGP also withdraws routes. When a session drops, everything learned through it is removed from the table automatically, and traffic shifts to whatever is left. A static route cannot do that — it points at a dead tunnel exactly as confidently as a live one, and keeps sending traffic into a hole until a human notices.

So BGP is not an optional extra layered on top of HA VPN. It is the mechanism that makes the HA real. Two tunnels with static routes gives you two tunnels and an outage.

What "high availability" actually requires

Google's wording is specific: for the 99.99% SLA "there must be a tunnel from each of the two interfaces on the HA VPN gateway to the corresponding interfaces on the peer gateway." Two interfaces, two tunnels, two peer addresses.

And a single active interface does not get a reduced SLA — it gets none. That is worth knowing before someone builds one tunnel and calls it done.

One thing worth getting exactly right, because the loose version is everywhere: Classic VPN is not deprecated — but BGP on it is. Google deprecated dynamic routing for Classic VPN tunnels on 1 August 2025. Creating a Classic VPN tunnel that uses BGP is no longer supported; tunnels that already exist “will continue to function, but without an availability SLA”. Static routing on Classic VPN remains supported and keeps its 99.9% SLA.

Which means the specific thing this week is about cannot be built on Classic VPN any more. Not slower, not unsupported-but-working — you cannot create it. HA VPN is the only way to get a BGP session with Google, and that is a better reason to use it than any SLA table.

Architecture and Workflow

A hub VPC and a branch VPC joined by two HA VPN tunnels. A Cloud Router on each side runs BGP over the tunnels on different ASNs, so each VPC learns the other's subnet rather than being told it. When one tunnel drops, BGP withdraws the routes learned through it in about three seconds and the surviving tunnel carries the traffic.
The branch VPC stands in for a remote site. The protocols do not care whether the peer is another VPC, on-premises hardware, or another cloud.
  1. A second VPC and subnet are created to stand in for a remote site, on a range that cannot overlap the hub's.
  2. An HA VPN gateway is created on each side, each with two interfaces and their own public IPs.
  3. A Cloud Router is created on each side, on different ASNs.
  4. Four tunnel resources are declared — two logical tunnels, each side declaring its own end — pairing interface 0 to interface 0 and 1 to 1.
  5. Each router gets an interface and a BGP peer per tunnel, addressed on link-local 169.254.x.x/30 ranges.
  6. The sessions establish, each side advertises its own subnet, and both route tables fill themselves in.
  7. One BGP peer is disabled. The routes learned through it are withdrawn, and the surviving tunnel carries the prefix.

Design decisions worth stating

Different ASNs on each side. Identical ASNs would make this an iBGP session, which does not re-advertise learned routes the way this topology needs.

The branch is a VPC, not another cloud. The roadmap called this week "cross-cloud", and the honest version is narrower: Google documents HA VPN topologies for on-premises gateways, Google Cloud VPCs and Compute Engine VMs, and no topology for a peer belonging to another cloud. A peer in the AWS lab is a genuinely different week, and it waits for one both windows can build together.

Prerequisites

  • A hub VPC (Week 04) with at least one subnet.
  • Keyless CI through Workload Identity Federation (Week 02).
  • Pipeline permissions on the platform folder. compute.networkAdmin was already there and covers networks, routers and VPN gateways. It does not cover firewall writes or instance creation, so this week added compute.securityAdmin and compute.instanceAdmin.v1.
  • Non-overlapping address space. BGP advertises these ranges; an overlap makes the route ambiguous.
Repository structure for week 13, showing terraform with main.tf and variables.tf, the scripts directory with measure-failover.py, gen-psk.sh and validate.sh, and docs
Two VPCs, two gateways, two routers, four tunnels — and one script that breaks a BGP peer and puts it back.

Implementation

What was deployed

WhatHow manyIts job
Terraform resource blocks16Expanding to 22 applied instances, because the tunnels and BGP peers use for_each
HA VPN gateways2Two interfaces each, one per side
Cloud Routers2ASN 64512 and 64513
VPN tunnels4Two logical tunnels; each side declares its own end, and each bills
BGP interfaces and peers8One pair per tunnel per side, on link-local /30s
Probe VMs2Something for a route to lead to

Step 1 — Gateways and routers

resource "google_compute_router" "hub" {
  name    = "hub-router"
  network = data.google_compute_network.hub.id
  region  = var.region

  bgp {
    asn = var.hub_asn        # 64512 — must differ from the branch
  }
}

Validation: both gateways report two interfaces with distinct public IPs.

Step 2 — Four tunnels for two logical tunnels

resource "google_compute_vpn_tunnel" "hub" {
  for_each = { t0 = 0, t1 = 1 }

  name                  = "hub-to-branch-${each.key}"
  vpn_gateway           = google_compute_ha_vpn_gateway.hub.id
  peer_gcp_gateway      = google_compute_ha_vpn_gateway.branch.id
  vpn_gateway_interface = each.value
  router                = google_compute_router.hub.id
  ike_version           = 2
}

Validation: all four report ESTABLISHED once this is applied — Step 5.

Step 3 — One BGP session per tunnel

Each router gets one interface and one peer per tunnel, addressed on link-local 169.254.x.x/30 ranges. No workload ever uses those; they exist so the routers have somewhere to talk.

resource "google_compute_router_peer" "hub" {
  for_each = local.bgp

  name            = "hub-peer-${each.key}"
  router          = google_compute_router.hub.name
  interface       = google_compute_router_interface.hub[each.key].name
  peer_ip_address = each.value.branch
  peer_asn        = var.branch_asn
}

Validation: two sessions reach Established on each router — one per tunnel. One is not a degraded version of two; it is the configuration with no SLA.

Step 4 — Deploy it

HCP Terraform workspace run list for gcp-week-13 showing five runs: three applied and two errored, including the destroy run
Five runs: three applied, two errored. The two failures are in the Troubleshooting section — both were this lab's own controls.

Step 5 — What came up

Now the tunnels exist. Four resources, two logical tunnels, all four established.

Terminal output listing four VPN tunnels all with status ESTABLISHED and detailed status Tunnel is up and running, plus the two gateways and their networks
Four tunnel resources for two logical tunnels — and four is what bills.

Step 6 — The route nobody wrote

And with the sessions up, each router has learned the other side’s subnet. This is the step worth pausing on: the prefix below is in the route table and in no .tf file.

Terminal output showing both BGP peers Established with one learned route each, two best routes to 10.30.0.0/24 via different next hops at equal priority, and a grep showing that CIDR appears in the Terraform only as a subnet variable
Two entries for one prefix, at equal priority. The grep is the point: 10.30.0.0/24 appears in the Terraform once, as a subnet — never as a route.

Validation and Evidence

TestExpected resultActual resultStatus
All four tunnels establishESTABLISHED4 of 4, "Tunnel is up and running"Pass
Both BGP sessions come upTwo EstablishedTwo, one learned route eachPass
The hub learns the branch prefixLearned, not configuredTwo entries for 10.30.0.0/24, equal priority, different next hopsPass
That prefix is not a static routeAbsent from the TerraformOne match, and it is a subnet CIDR variablePass
Routes withdraw when a peer dropsCount falls to one3.6s, 3.1s, 3.3s across three trialsPass
Routes return when it recoversCount returns to two3.1s, 3.3s, 3.4sPass
Final state restoredTwo routes, two sessionsTwo and twoPass
Terminal output from measure-failover.py showing a baseline of two routes and two established sessions, then three trials disabling a BGP peer, each showing the route withdrawn in around three seconds and restored in around three seconds
Three trials. Routes withdrawn in 3.1–3.6 seconds, restored in 3.1–3.4. Nobody edited a route table.

What the route count actually proves. Each tunnel advertises 10.30.0.0/24 independently, so two entries means both paths are live and one means BGP has withdrawn the dead one. That count is the failover signal — it is the thing a static route could never produce.

One honest caveat. Every assertion above was run against the live deployment and its output is in the figures. They were not run through validate.sh as a single script before teardown — the script exists for reproduction, but it was not executed end to end against this deployment, and this post does not claim it was.

Security and Cost

Security

  • The pre-shared key is generated, never typed. gen-psk.sh writes a random key into a gitignored tfvars file; no credential appears in this repository.
  • The probe VMs have no external addresses. They exist to be routed to, not reached from outside.
  • Firewall rules name the peer ranges explicitly rather than opening to anything that arrives over the tunnel. A tunnel being encrypted says nothing about what should be allowed through it.
  • Both VMs carry enable-oslogin and block-project-ssh-keys, because the organization's policies refuse to create them otherwise — see Troubleshooting.
  • No service account keys, as in every week of this lab.

Cost

ItemRateThis run
VPN tunnels × 4$0.05 / hour each06:00–17:00 → $1.99
Cloud Routers × 2no charge for the router itself$0
Probe VMs × 2e2-micro, us-central1, no external IP$0.17 charged, $0.17 free-tier discount → $0
Network Intelligence Centerresource hours$0.03, fully credited → $0
Total$1.99

Four, not two. This is the number to plan around. Two logical tunnels between two Google Cloud VPCs means a tunnel resource on each side, and each one bills. Left running, that is about $146 a month — against a $25/month budget alert on the whole account. This week is affordable only because it was destroyed the same day.

Reconciled against the posted charges in this lab’s own BigQuery billing export: $1.99, which matches the rate × runtime figure exactly. Two cautions worth more than the number. The same query run at 22:40 on the day of the deploy returned $1.76, because that day’s rows were still being written — a partially loaded export looks exactly like a settled one. And filtering it to cost > 0 hides the free-tier discount lines, which are negative rows, and overstates the week by the full price of the VMs.

Troubleshooting and Lessons

The pipeline could build the network but not the things in it

Problem: the apply created gateways, routers and all four tunnels, then failed with 403 on compute.firewalls.create and compute.instances.create.
Root cause: compute.networkAdmin covers networks, routers and VPN but not firewall writes; and Week 04 granted instanceAdmin on the workloads folder, not platform.
Lesson: a role named for a domain does not cover everything in that domain. The split between "the network" and "rules applied to the network" is a different role, and the same create-versus-modify boundary has now appeared in four separate weeks of this lab.

The organization refused to create the VMs

Problem: Error 412: Operation denied by org policy on both requireOsLogin and blockProjectSshKeys.
Root cause: Week 03 set those constraints and Week 04 promoted them to enforced. Any VM created afterwards must carry the matching metadata.
Why it had never happened before: the only other VM in this lab predates enforcement. It was created while the constraints were still in dry run, so nothing had ever been built under them. This was the first.
Lesson: a guardrail promoted after the fact is untested until something new arrives. The gap between "enforced" and "exercised" can be months.

The flag is --no-enabled, and there is no --disable

Problem: the failover script failed with unrecognized arguments: --disable.
Root cause: gcloud compute routers update-bgp-peer takes --enabled / --no-enabled.
Lesson: worth recording because the failure was safe by construction — the disable never happened, so nothing was left broken, and the restore in the finally block had nothing to undo. A script that degrades a control should be written so that failing early leaves the control intact.

Cleanup and Key Takeaways

Cleanup

terraform destroy removed everything this week created and left Week 04's hub VPC in place. Verified against the cloud rather than against state: zero tunnels, zero routers, zero gateways, zero instances, and the branch VPC gone.

Unlike Week 12's Cloud KMS key, nothing here survives a destroy and nothing keeps billing afterwards.

Key takeaways

  1. BGP is what makes HA VPN highly available. Two tunnels with static routes gives you two tunnels and an outage. The withdrawal behaviour — measured here at 3.1–3.6 seconds — is the entire difference.
  2. A single active interface carries no SLA. Not a reduced one. None.
  3. Four tunnel resources, not two. Each side declares its own end and each bills, so the hourly rate is double what the topology diagram suggests.
  4. A learned route is invisible in your code, which is the point and also the thing to remember when debugging. If a prefix is reachable and you cannot find it in your Terraform, that is BGP working.
  5. Enforced is not exercised. Two org policies set months ago blocked this build on their first real test, because the only VM that existed predated them.

Repository

week-13-hybrid-connectivity — Terraform, the failover measurement, and the lab guide.

References

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent