Week 4 — Shared VPC on Google Cloud: The Identity I Built Could Not Do This Week

GCP Platform Engineering Lab · Week 4 of 58

13 September 2026 Shared VPC · Cloud NGFW · Terraform ~$0 · and four choices keep it that way

📋 In This Post

Why — The Problem This Solves
What You Need to Know — Skills & Tools
Architecture — How It Fits Together
How We Built It — Step by Step
Challenges — What Actually Went Wrong
Security — Controls at Every Layer
Cost
Cleanup
References
Key Takeaways

Why — The Problem This Solves

This week was number twelve until nine days ago. It moved, and the reason is a dependency the original order had backwards.

Week 3 finished with three constraints sitting in dry run: requireOsLogin, blockProjectSshKeys, disableSerialPortAccess. A dry-run policy is evaluated on every request and logged when it would have denied — which means with no Compute Engine resource anywhere in the organization, it evaluates nothing and logs nothing. Those three could never be promoted on evidence, because there was no evidence to be had. Week 3's unfinished business was parked behind eight weeks of more control plane.

So this week exists to make the first request those rules can judge.

And the shape of the thing that makes it matters. One project owning the network, another owning the workload, and nothing owning both.

What You Need to Know — Skills & Tools

Shared VPC splits ownership of the network from ownership of what runs on it. A host project owns the VPC and its subnets. Service projects are attached to it and place resources into those subnets without owning any network of their own. The platform team holds addressing and firewall policy; application teams hold their workloads.

Custom mode, not auto mode. An auto-mode VPC creates a subnet in every region Google has — now, and in every region added later — each with a range nobody chose. That is an address plan decided by Google's expansion schedule. Custom mode means the only ranges that exist are the ones written down.

Hierarchical firewall policies attach to organization and folder nodes, and are evaluated before any VPC firewall rule. An allow or deny at that level is final; goto_next is what hands the decision downward. A policy written entirely in allow/deny at the top takes every firewall decision in the organization away from the teams below it.

One thing worth checking rather than assuming, because the opposite is a plausible thing to write: the product is now Cloud Next Generation Firewall, and VPC firewall rules are not deprecated. The reference documentation presents them and network firewall policies as coexisting, and recommends neither over the other.

Architecture — How It Fits Together

The network hub project has existed since Week 1 and had never had a VPC in it — Week 1 set auto_create_network = false, so not even the permissive default network Google would otherwise create.

Google Cloud console showing the VPC networks list for the network hub project, empty — The host project before this week. Empty, deliberately, since Week 1 — captured before the apply, because an empty list is exactly the state that stops existing the moment you deploy.

The hub VPC detail page showing custom subnet mode with the dev and prod subnets and their ranges — Two subnets, both written down. `prod` is attached to nothing and exists anyway — reserving the range is the point, because an address plan made in a hurry six months from now takes whatever is free rather than what was intended.

How We Built It — Step by Step

The build order was forced by something I did not anticipate, and it is the most interesting thing in the week.

The CI identity could not do this week

Reading state before writing code — which this lab now does as a rule — turned up that tf-apply held six roles covering folders, projects, service usage, service accounts and organization policy, and not one compute permission. It could not create a network, enable Shared VPC hosting, write a firewall policy or create an instance.

The obvious fix is to grant those roles in this week's configuration, next to the resources that need them. That is the one thing this lab will not do. Week 2 established the rule and the reason: an identity must not manage its own grants. A configuration applied by tf-apply that also widens tf-apply is one where CI can give itself anything it later decides it wants, and no reviewer sees a difference between that and a legitimate feature.

So the grants went into Week 2's configuration — the human-run layer, applied from a person's credentials — and this week consumes them. That is the two-layer split working rather than an obstacle to route around. It also means every widening of CI is a separate, separately-reviewed act instead of a line buried in whichever week happened to need it.

Scope where scope was possible: compute.networkAdmin and compute.instanceAdmin.v1 on folders, not the organization. One role resisted. compute.xpnAdmin cannot be folder-scoped — designating a project as a Shared VPC host is checked at the organization — so it is now the widest permission this lab has given CI, and that is worth stating plainly rather than glossing.

The Shared VPC page for the host project showing the attached service project — Host enablement and attachment are two separate operations. A host with no attachments lends nothing.

The instance, and why every attribute of it is a cost decision

One e2-micro, us-central1, pd-standard 10 GB, and no external IP. None of those is a preference:

e2-micro — the only machine type in the Always Free tier
us-central1 — the free tier covers us-west1, us-central1 and us-east1 only
pd-standard — the free 30 GB-month allowance is standard disk. pd-balanced is the default and bills from the first byte
no external IP — an external IPv4 address is billed per hour whether traffic flows or not, and is not in the free tier

Get any one wrong and a free week quietly becomes a billed one. The validation script asserts the absent external address as a cost check rather than a connectivity one, because an accidental external IP is a bill that appears without anything appearing broken.

VM instances list showing dev-app-01 with internal IP 10.20.0.2 and an empty External IP column — Internal IP **10.20.0.2**, External IP **empty**. The address comes from a subnet in a different project — that one value is the whole of Shared VPC.

Then the point of the week

With a Compute resource finally in existence, Week 3's three constraints had something to evaluate and were promoted out of dry run. The effective policy, at three scopes, evaluated by Google rather than asserted by me:

constraint                                 org    dev    prod
compute.managed.requireOsLogin             True   True   True
compute.managed.blockProjectSshKeys        True   True   True
compute.managed.disableSerialPortAccess    True   False  True

One row, one column. Week 3 wrote a deliberate exception at workloads/dev for serial console access — because the thing the constraint defends against and the thing a developer needs when a machine will not boot are the same thing. Until this week that was a claim. Now it is inheritance everywhere else and an override exactly where it was written.

Challenges — What Actually Went Wrong

1. Creating a firewall policy and attaching it are different roles

The policy created cleanly. The association was refused:

Error 403: Required 'compute.organizations.setFirewallPolicy'
permission for 'folders/...', forbidden

The error names a permission and no role that grants it. Reading both role definitions settled it:

roles/compute.orgSecurityPolicyAdmin    firewallPolicies.create/update/use
                                        NOT organizations.setFirewallPolicy
roles/compute.orgSecurityResourceAdmin  organizations.setFirewallPolicy

The split is reasonable once stated — a policy attached to nothing is inert, so attachment is the act that actually changes behaviour. Nothing in the names says so.

2. A resource owned by no project still needs a project

A hierarchical firewall policy hangs off a folder. The Compute API is client-based, meaning it attributes calls to the caller's quota project rather than the resource's — and with none set:

Error 404: The resource 'projects/null' was not found

Which reads like a missing resource rather than a missing header. The fix is user_project_override plus billing_project on the provider; the provider does not send X-Goog-User-Project unless the override is true, so setting the billing project alone does nothing.

The bootstrap hit the identical thing on the Budgets API in August. Knowing that did not make this recognisable, because the error text is different and points somewhere else.

3. And that fix cascades

Once every call is attributed to the host project, that project needs APIs enabled that it does not itself use. Three of them, discovered one failed apply at a time: cloudresourcemanager, then serviceusage, then cloudbilling.

Each is chicken-and-egg — the call that enables the API is itself attributed to the project that needs it — so the first enablement of each had to be done by hand before Terraform could manage it.

4. A billing account caps how many projects it will fund

Project creation failed with Cloud billing quota exceeded at five linked projects. The number is not documented anywhere I could find; it surfaces as a failure.

Freeing a slot meant unlinking an old sandbox project — and the organization administrator could not do it, because that project sits outside the organization. Only the identity that owns it could. A useful reminder that "I am the admin" is scoped to a hierarchy, and things outside that hierarchy do not care.

5. Enforcement lag is a property of the policy system, not of one constraint

Week 3 measured 75–100 seconds between writing a policy and it actually refusing anything, on a custom constraint. I assumed that was specific to custom constraints.

It is not. The first attempt to violate blockProjectSshKeys — a Google-managed constraint, a completely different generation — succeeded, one to two minutes after promotion. The same request minutes later was refused. Two different constraint types, same window.

Worth internalising before you test a control and conclude it does not work.

6. Enforcement is not retroactive, and Google says so in the refusal

The denial text carries a caveat that deserves more attention than it gets:

Enforcing this constraint does not affect existing VMs where block-project-ssh-keys is already set to false; they will retain access unless their metadata is updated.

A machine weakened before the rule went on stays weakened. This is the same property that put governance before workloads in this roadmap in the first place — organization policy is evaluated at request time, and a constraint added later does not go back and fix what already exists.

Security — Controls at Every Layer

No external address, and no bastion either. SSH arrives through Identity-Aware Proxy from 35.235.240.0/20, which is the one ingress path the firewall policy leaves open. IAP brokers the connection, authenticates it against IAM and logs it. A bastion host would be a second machine to patch, and a public address to defend.

Default-deny at the folder, before any VPC rule. The hierarchical policy governs every project under workloads including ones that do not exist yet. A VPC firewall rule protects one network; this protects a branch of the hierarchy.

networkUser granted on the subnet, not the project. At project scope, the service project could place resources in any subnet the host owns — including prod. Scoped to one subnet, dev can use dev.

The widening of CI is visible. Six roles became eleven, in a configuration a human applies, with the reasoning written next to each one. The alternative — granting them where they were needed — would have been shorter, would have worked, and would have made CI its own privilege administrator.

One thing I will flag rather than dress up: compute.xpnAdmin is organization-wide because the API gives no choice. It permits designating any project in the organization as a Shared VPC host and attaching any project to it. It does not permit changing the networks themselves. That is the honest boundary.

Cost

Approximately zero, and the roadmap was wrong to promise otherwise. This was pencilled in as the first week with a real bill; a careful design keeps it inside the Always Free tier, which is more useful to a reader than a number.

Free: Shared VPC, subnets, hierarchical firewall policies, the extra project, Private Google Access, flow logs at this volume. Free because chosen carefully: the machine type, the region, the disk type and the absence of an external IP.

The one thing to watch is that the free tier is one e2-micro per billing account per month, not per project. A second instance anywhere bills normally.

Cleanup

This week is meant to stay running — it is the thing later weeks build on, and it costs nothing to leave. What matters more than teardown is what the teardown would not undo: the compute grants in Week 2 persist, because they define what CI is rather than what this week built.

If you do tear it down, delete the instance before the subnet and the subnet before the network, and expect the service project detachment to need the same xpnAdmin that attached it.

References

Shared VPC overview — host and service project model.
Hierarchical firewall policies — and the detail that in a Shared VPC, a VM is governed by the host project's hierarchical rules.
Cloud NGFW policy types — confirms VPC firewall rules are not deprecated.
IAP TCP forwarding — the 35.235.240.0/20 range.
Always Free tier — every cost decision here traces to this page.

Not answered anywhere I looked, and measured instead: which role attaches a firewall policy to a folder; that a folder-scoped policy needs a quota project; that the override then makes the quota project accumulate unused APIs; the billing account's project cap; that enforcement lag applies to managed constraints too; and that an organization administrator cannot read a hierarchical firewall policy without compute.orgFirewallPolicyUser.

Key Takeaways

An identity that can widen itself is not a boundary. The most useful thing this week did was fail early, because the failure landed in the human-run layer where it belonged.
Scope to a folder where the API allows it, and say so when it does not. Three of four compute grants narrowed. The fourth could not, and pretending otherwise would be worse than admitting it.
Every attribute of a VM is a cost decision. Machine type, region, disk type, external address. Four values between free and billed, none of which announce themselves.
Test a control twice before believing it. A guardrail that lets something through may simply not have taken effect yet — and the wrong conclusion sends you editing correct code.
Governance before workloads is not pedantry. Enforcement does not reach backwards. Anything created before the rule stays as it was.

Next week the projects stop being hand-written. Four exist now, each created a slightly different way, which is exactly the point at which a factory starts being worth having.

The identity I built could not do this week