📋 In This Post
Lab Overview
The problem. Building an encrypted tunnel between two networks is the easy part. Both ends agree on ciphers, the tunnel comes up, and a dashboard turns green. Then a packet leaves a machine on one side headed for an address on the other, and the network has to answer a question nobody has answered yet: does this go down the tunnel, or out to the internet? Something has to put that answer in a route table. Type it by hand and it is correct until the day the other side changes — or until the tunnel dies, and the route keeps pointing at it with total confidence.
The objective. Build HA VPN between two VPCs, let BGP populate both route tables, then break one tunnel and measure how long the network takes to route around it.
Expected outcome. Two route tables neither side typed, each holding the other's subnet twice — once per tunnel. Kill one tunnel and the count halves, traffic keeps flowing, and no human touches anything. If the prefix only appears once, the second tunnel is not carrying it and the redundancy is nominal.
Deployment method. Terraform on HCP Terraform agents with no stored credentials — the workspace mints an OIDC token per run and exchanges it for a short-lived Google token, with separate identities for plan and apply. Built in Week 02; every week since rides on it.
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| HA VPN | Carries the traffic between the two VPCs | Google Cloud's managed IPsec VPN. A gateway has two interfaces with their own public IPs; using both is what the availability SLA is defined on. |
| Cloud Router | Runs BGP and programs what it learns into the VPC | A managed BGP speaker. It holds your ASN and decides what the route table says — it is not in the data path, so traffic never flows through it. |
| BGP | Fills both route tables, and empties them when a tunnel dies | The protocol networks use to tell each other which address ranges they can reach. The same one that runs the public internet between ISPs. |
| VPC and subnets | The two networks being joined | A VPC is a private network; a subnet is a range inside it tied to one region. Custom mode means no subnet exists until you declare it. |
| Compute Engine | Two probe VMs, one per side | Virtual machines. Here they are instruments rather than the subject — something has to exist at each end for a route to lead to. |
| Terraform & HCP Terraform | Declares all of it and applies it remotely | Terraform describes infrastructure as code; HCP runs it on hosted agents, authenticating per run through Workload Identity Federation so no key is stored. |
Static routes versus BGP, and why it is not a style choice
A route table says: for addresses in this range, send the packet that way. For your own subnets it fills itself in. For anything beyond a tunnel, something has to add the entry.
You can type it — anything for 10.30.0.0/24, send down
tunnel 1. Or the two routers can tell each other, continuously, which
ranges they can reach. That conversation is BGP, and each side identifies itself
with an ASN; there is a private range, 64512–65534, for
cases like this.
The difference that matters is not convenience. BGP also withdraws routes. When a session drops, everything learned through it is removed from the table automatically, and traffic shifts to whatever is left. A static route cannot do that — it points at a dead tunnel exactly as confidently as a live one, and keeps sending traffic into a hole until a human notices.
So BGP is not an optional extra layered on top of HA VPN. It is the mechanism that makes the HA real. Two tunnels with static routes gives you two tunnels and an outage.
What "high availability" actually requires
Google's wording is specific: for the 99.99% SLA "there must be a tunnel from each of the two interfaces on the HA VPN gateway to the corresponding interfaces on the peer gateway." Two interfaces, two tunnels, two peer addresses.
And a single active interface does not get a reduced SLA — it gets none. That is worth knowing before someone builds one tunnel and calls it done.
One thing worth getting exactly right, because the loose version is everywhere: Classic VPN is not deprecated — but BGP on it is. Google deprecated dynamic routing for Classic VPN tunnels on 1 August 2025. Creating a Classic VPN tunnel that uses BGP is no longer supported; tunnels that already exist “will continue to function, but without an availability SLA”. Static routing on Classic VPN remains supported and keeps its 99.9% SLA.
Which means the specific thing this week is about cannot be built on Classic VPN any more. Not slower, not unsupported-but-working — you cannot create it. HA VPN is the only way to get a BGP session with Google, and that is a better reason to use it than any SLA table.
Architecture and Workflow
- A second VPC and subnet are created to stand in for a remote site, on a range that cannot overlap the hub's.
- An HA VPN gateway is created on each side, each with two interfaces and their own public IPs.
- A Cloud Router is created on each side, on different ASNs.
- Four tunnel resources are declared — two logical tunnels, each side declaring its own end — pairing interface 0 to interface 0 and 1 to 1.
- Each router gets an interface and a BGP peer per tunnel, addressed on link-local
169.254.x.x/30ranges. - The sessions establish, each side advertises its own subnet, and both route tables fill themselves in.
- One BGP peer is disabled. The routes learned through it are withdrawn, and the surviving tunnel carries the prefix.
Design decisions worth stating
Different ASNs on each side. Identical ASNs would make this an iBGP session, which does not re-advertise learned routes the way this topology needs.
The branch is a VPC, not another cloud. The roadmap called this week "cross-cloud", and the honest version is narrower: Google documents HA VPN topologies for on-premises gateways, Google Cloud VPCs and Compute Engine VMs, and no topology for a peer belonging to another cloud. A peer in the AWS lab is a genuinely different week, and it waits for one both windows can build together.
Prerequisites
- A hub VPC (Week 04) with at least one subnet.
- Keyless CI through Workload Identity Federation (Week 02).
- Pipeline permissions on the platform folder.
compute.networkAdminwas already there and covers networks, routers and VPN gateways. It does not cover firewall writes or instance creation, so this week addedcompute.securityAdminandcompute.instanceAdmin.v1. - Non-overlapping address space. BGP advertises these ranges; an overlap makes the route ambiguous.
Implementation
What was deployed
| What | How many | Its job |
|---|---|---|
| Terraform resource blocks | 16 | Expanding to 22 applied instances, because the tunnels and BGP peers use for_each |
| HA VPN gateways | 2 | Two interfaces each, one per side |
| Cloud Routers | 2 | ASN 64512 and 64513 |
| VPN tunnels | 4 | Two logical tunnels; each side declares its own end, and each bills |
| BGP interfaces and peers | 8 | One pair per tunnel per side, on link-local /30s |
| Probe VMs | 2 | Something for a route to lead to |
Step 1 — Gateways and routers
resource "google_compute_router" "hub" {
name = "hub-router"
network = data.google_compute_network.hub.id
region = var.region
bgp {
asn = var.hub_asn # 64512 — must differ from the branch
}
}
Validation: both gateways report two interfaces with distinct public IPs.
Step 2 — Four tunnels for two logical tunnels
resource "google_compute_vpn_tunnel" "hub" {
for_each = { t0 = 0, t1 = 1 }
name = "hub-to-branch-${each.key}"
vpn_gateway = google_compute_ha_vpn_gateway.hub.id
peer_gcp_gateway = google_compute_ha_vpn_gateway.branch.id
vpn_gateway_interface = each.value
router = google_compute_router.hub.id
ike_version = 2
}
Validation: all four report ESTABLISHED once this is applied — Step 5.
Step 3 — One BGP session per tunnel
Each router gets one interface and one peer per tunnel, addressed on
link-local 169.254.x.x/30 ranges. No workload ever uses those; they
exist so the routers have somewhere to talk.
resource "google_compute_router_peer" "hub" {
for_each = local.bgp
name = "hub-peer-${each.key}"
router = google_compute_router.hub.name
interface = google_compute_router_interface.hub[each.key].name
peer_ip_address = each.value.branch
peer_asn = var.branch_asn
}
Validation: two sessions reach Established on
each router — one per tunnel. One is not a degraded version of two; it is
the configuration with no SLA.
Step 4 — Deploy it
Step 5 — What came up
Now the tunnels exist. Four resources, two logical tunnels, all four established.
Step 6 — The route nobody wrote
And with the sessions up, each router has learned the other side’s
subnet. This is the step worth pausing on: the prefix below is in the
route table and in no .tf file.
10.30.0.0/24 appears in the Terraform once, as a subnet — never as a route.Validation and Evidence
| Test | Expected result | Actual result | Status |
|---|---|---|---|
| All four tunnels establish | ESTABLISHED | 4 of 4, "Tunnel is up and running" | Pass |
| Both BGP sessions come up | Two Established | Two, one learned route each | Pass |
| The hub learns the branch prefix | Learned, not configured | Two entries for 10.30.0.0/24, equal priority, different next hops | Pass |
| That prefix is not a static route | Absent from the Terraform | One match, and it is a subnet CIDR variable | Pass |
| Routes withdraw when a peer drops | Count falls to one | 3.6s, 3.1s, 3.3s across three trials | Pass |
| Routes return when it recovers | Count returns to two | 3.1s, 3.3s, 3.4s | Pass |
| Final state restored | Two routes, two sessions | Two and two | Pass |
What the route count actually proves. Each tunnel advertises
10.30.0.0/24 independently, so two entries means both paths are
live and one means BGP has withdrawn the dead one. That count is the
failover signal — it is the thing a static route could never produce.
One honest caveat. Every assertion above was
run against the live deployment and its output is in the figures. They were not
run through validate.sh as a single script before teardown —
the script exists for reproduction, but it was not executed end to end against
this deployment, and this post does not claim it was.
Security and Cost
Security
- The pre-shared key is generated, never typed.
gen-psk.shwrites a random key into a gitignored tfvars file; no credential appears in this repository. - The probe VMs have no external addresses. They exist to be routed to, not reached from outside.
- Firewall rules name the peer ranges explicitly rather than opening to anything that arrives over the tunnel. A tunnel being encrypted says nothing about what should be allowed through it.
- Both VMs carry
enable-osloginandblock-project-ssh-keys, because the organization's policies refuse to create them otherwise — see Troubleshooting. - No service account keys, as in every week of this lab.
Cost
| Item | Rate | This run |
|---|---|---|
| VPN tunnels × 4 | $0.05 / hour each | 06:00–17:00 → $1.99 |
| Cloud Routers × 2 | no charge for the router itself | $0 |
| Probe VMs × 2 | e2-micro, us-central1, no external IP | $0.17 charged, $0.17 free-tier discount → $0 |
| Network Intelligence Center | resource hours | $0.03, fully credited → $0 |
| Total | $1.99 |
Four, not two. This is the number to plan around. Two logical tunnels between two Google Cloud VPCs means a tunnel resource on each side, and each one bills. Left running, that is about $146 a month — against a $25/month budget alert on the whole account. This week is affordable only because it was destroyed the same day.
Reconciled against the posted charges in this lab’s own
BigQuery billing export: $1.99, which matches the rate
× runtime figure exactly. Two cautions worth more than the number. The
same query run at 22:40 on the day of the deploy returned $1.76, because that
day’s rows were still being written — a partially loaded export
looks exactly like a settled one. And filtering it to cost > 0
hides the free-tier discount lines, which are negative rows, and overstates
the week by the full price of the VMs.
Troubleshooting and Lessons
The pipeline could build the network but not the things in it
Problem: the apply created gateways, routers and all four
tunnels, then failed with 403 on compute.firewalls.create and
compute.instances.create.
Root cause: compute.networkAdmin covers networks,
routers and VPN but not firewall writes; and Week 04 granted
instanceAdmin on the workloads folder, not
platform.
Lesson: a role named for a domain does not cover everything in
that domain. The split between "the network" and "rules applied to the network"
is a different role, and the same create-versus-modify boundary has now appeared
in four separate weeks of this lab.
The organization refused to create the VMs
Problem: Error 412: Operation denied by org policy
on both requireOsLogin and blockProjectSshKeys.
Root cause: Week 03 set those constraints and Week 04 promoted
them to enforced. Any VM created afterwards must carry the matching metadata.
Why it had never happened before: the only other VM in this lab
predates enforcement. It was created while the constraints were still
in dry run, so nothing had ever been built under them. This was the first.
Lesson: a guardrail promoted after the fact is untested until
something new arrives. The gap between "enforced" and "exercised" can be
months.
The flag is --no-enabled, and there is no --disable
Problem: the failover script failed with
unrecognized arguments: --disable.
Root cause: gcloud compute routers update-bgp-peer
takes --enabled / --no-enabled.
Lesson: worth recording because the failure was safe by
construction — the disable never happened, so nothing was left broken, and
the restore in the finally block had nothing to undo. A script that
degrades a control should be written so that failing early leaves the control
intact.
Cleanup and Key Takeaways
Cleanup
terraform destroy removed everything this week created and left
Week 04's hub VPC in place. Verified against the cloud rather than against
state: zero tunnels, zero routers, zero gateways, zero instances, and the
branch VPC gone.
Unlike Week 12's Cloud KMS key, nothing here survives a destroy and nothing keeps billing afterwards.
Key takeaways
- BGP is what makes HA VPN highly available. Two tunnels with static routes gives you two tunnels and an outage. The withdrawal behaviour — measured here at 3.1–3.6 seconds — is the entire difference.
- A single active interface carries no SLA. Not a reduced one. None.
- Four tunnel resources, not two. Each side declares its own end and each bills, so the hourly rate is double what the topology diagram suggests.
- A learned route is invisible in your code, which is the point and also the thing to remember when debugging. If a prefix is reachable and you cannot find it in your Terraform, that is BGP working.
- Enforced is not exercised. Two org policies set months ago blocked this build on their first real test, because the only VM that existed predated them.
Repository
week-13-hybrid-connectivity — Terraform, the failover measurement, and the lab guide.
References
- Cloud VPN overview — HA VPN, and Classic VPN's continuing 99.9% SLA
- HA VPN topologies — what the 99.99% SLA requires, and which peers are documented
- Cloud Router overview — BGP, ASNs and route advertisement
Comments