📋 In This Post
Why — The Problem This Solves
Five projects existed before this week, and all five carry the same three labels for week, environment and owner. It would be easy to call that a working convention.
It is not a convention. It is a coincidence with a short history. Three of those projects were written in one sitting in Week 1, the fourth copied them in Week 4, and the fifth is the one this week built. Nothing in the organization required any of it. A project created in the console tomorrow would carry no labels, because the console does not ask for any, and nothing would notice.
That is the gap a factory closes, and the reason it is worth building at the point where project creation is easy rather than hard. On AWS, accounts are rationed and expensive, so the vending machine exists to make a difficult thing possible. On Google Cloud, projects are free and disposable — which means four people will make four of them, four different ways, before anyone thinks to standardise.
So the week is two halves that need each other: a module that makes the right project, and a rule that refuses the wrong one. A factory that is merely available loses to a deadline.
What You Need to Know — Skills & Tools
Labels and tags are not two words for the same thing, and this week turns on the difference.
A label is free text. Anyone who can edit a project can
write env = whatever-they-like. Labels are queryable, they land in
the billing export, and they cannot be used to set conditions on
policies. They answer what did this cost.
A tag is a first-class resource with IAM on it and a closed
set of permitted values. Tags can be referenced by organization policy
conditions — resource.matchTag('ORG/environment', 'dev')
scopes a policy to exactly the resources carrying it. They answer which
rules apply to this.
The factory sets both, because they are not redundant. And the distinction matters more than it looks: I spent the first half of this week designing around a rule that read labels, and the second half discovering it could not.
One more thing worth knowing before you plan a week around it:
cloudresourcemanager.googleapis.com/Project is supported
for custom constraints at Preview, and the reference that
lists it stops there. It does not enumerate the fields a condition may read.
Hold that thought.
Architecture — How It Fits Together
The shape is small, and deliberately so. A reusable module, one organization tag key with two permitted values, and a custom constraint sitting above both.
organization
├── tagKeys/environment values: dev, prod (IAM-controlled, closed)
└── custom.requireProjectNamingAndParent DRY RUN
projectId must start with the org prefix
parent must be a folder
workloads/dev
└── katta698-gcp-dev-app-02 built by modules/project
labels week=05, env=dev, managed-by=terraform
tag environment=dev
essential contact · no default network · deletion_policy PREVENT
What the module refuses to make optional
This is the part worth copying. The module has no
auto_create_network toggle. Labels are always set. An essential
contact is always set. deletion_policy is stated rather than
inherited from the provider default.
A factory whose every control has a switch produces whatever the caller asked for, which is precisely what it was built to replace. The one thing it does leave optional is a deletion lien, because most projects should be deletable — a lab that cannot tear down its own work accumulates cost and confusion.
Essential contacts deserve their own sentence. Google sends security bulletins, deprecation warnings and suspension notices to Essential Contacts and to nobody else. A project with none is a project where the notice that a credential leaked goes to an address that does not exist. It is easy, it is invisible when missing, and nothing breaks until the one day it matters — which is the exact profile of a control that belongs in a factory rather than in a checklist.
How We Built It — Step by Step
First, how anything here gets deployed at all, because the rest of this section assumes it.
Nothing in this lab is built by hand. The Terraform is applied by
HCP Terraform, remotely, running as the tf-apply
service account Week 2 created. There is no Google Cloud credential on the
machine that starts the run: HCP mints an OIDC token per run, Google's
Security Token Service exchanges it, and the short-lived credential that comes
back never touches disk.
So every console screenshot below is evidence of what a Terraform run produced, never a thing that was clicked. The apply comes first and the resources follow from it.
With that established, here is the order it actually happened in, including the part where it stopped.
1. Reading state first, and finding the wall
The pre-build read is a rule in this lab, and this week it paid immediately. The billing account was at its project cap. Five linked projects, five allowed, nothing spare.
$ gcloud billing projects list --billing-account=...
katta698-gcp-net-hub
katta698-gcp-lab-seed
katta698-gcp-logging
katta698-gcp-security
katta698-gcp-dev-app-01 <- five linked, five allowed
$ terraform apply
Error: Cloud billing quota exceeded
Which is a particular kind of funny when the week's subject is a machine for making projects. Raising the cap is a Console form that Google may attach a payment to — not an API, not something Terraform reaches.
So Week 4's service project was deleted to free the slot. Its VM, its Shared VPC attachment and its subnet bindings went with it. The hub VPC, both subnets and the hierarchical firewall policy were kept — they occupy no project slot, and Weeks 13 to 19 build on them. The post describing that week is already published and its screenshots are committed, so the evidence outlives the infrastructure.
2. The CI identity could not build this week either
This is now the pattern rather than the exception. tf-apply
could create a project and could not create a tag key, bind a tag, or set an
essential contact. Four grants were needed.
None of them went into this week's configuration. An identity must
not manage its own grants — a config applied by
tf-apply that also widens tf-apply lets CI give itself
anything it later decides it wants, and no reviewer sees a difference between
that and a feature. They went into Week 2, the human-run layer, applied from a
person's credentials.
3. The tag key, and the module
An organization-level key with exactly two permitted values. The module is
handed a value ID and cannot construct one — which is what stops a caller
passing env = "prod-ish" and creating a tag value that no policy
reads, leaving a project that looks tagged while sitting outside every rule
expressed in terms of the tag.
environment key and its two permitted values. The
closed set is the control: the module is handed a value ID and has no way to
create one, so env = "prod-ish" cannot become a tag that no
policy reads.Then one project, built through the module, into the slot Week 4 gave up. Project creation took 3m20s, which is consistent with Week 1 and worth planning around.
environment : dev tag, and a location of
jayanthkatta.com › … › dev — the two things
the constraint can read, and the folder it had to be in.
4. And then the constraint was rejected
The rule was supposed to be: a project must carry the mandatory labels or it is refused. Google answered:
Error 400: Request has invalid values
That is the entire message. It does not name the field, the reason, or which of the several things in the request it objected to.
So I probed it. One constraint at a time, each with a single expression, recording which Google accepted:
resource.projectId ACCEPTED
resource.parent ACCEPTED
resource.labels REJECTED
resource.displayName REJECTED
A custom constraint on a project cannot read that project's
labels. The supported-services reference lists Project, at
Preview, and never publishes which of its fields are exposed. The design I had
spent the week on could never have worked, and no amount of reading would have
told me — only a probe did.
5. What was enforceable instead, and why it is better
Two fields remained, and the constraint rewritten onto them is arguably stronger than the one I set out to build:
resource.projectId.startsWith("katta698-gcp-") &&
resource.parent.startsWith("folders/")
A project ID is permanent and globally unique. It is never released for reuse, even after the project is deleted. So a name that breaks the standard cannot be corrected later — only abandoned. That makes it the one property of a project genuinely worth refusing at creation, in a way a label never was.
And a project created at the organization root rather than under a folder inherits none of the folder-level policy this lab spent three weeks building.
It sits in dry run, per the rule Week 3 established: a constraint that has never evaluated a request has no evidence behind it, and enforcing on no evidence is a guess wearing a control's clothes.
Challenges — What Actually Went Wrong
1. An error message that named nothing
Error 400: Request has invalid values, on a request containing
a resource type, two method types, an action type and a CEL expression. Any of
them could have been the problem.
The only way through was to make the request smaller until it succeeded.
A constraint on Project with the condition "true"
was accepted, which cleared the resource type and the method type in one step
and left the expression. Then one field per probe.
Worth internalising as a technique: when an error names nothing, stop reading it and start bisecting the request.
2. Creating a tag and attaching a tag are different roles
The tag key and its values created cleanly. The binding was refused with
Error 403: The caller does not have permission — no
permission named, no role suggested.
Reading both role definitions settled it:
roles/resourcemanager.tagAdmin creates keys and values,
NO tagValueBindings.create
roles/resourcemanager.tagUser HAS tagValueBindings.create
This is the second time in two weeks: Week 4 found that creating a hierarchical firewall policy and attaching it are also separate roles. The split is defensible — a tag value bound to nothing changes no behaviour, so binding is the act with consequences — but the names do not say so, and neither does the error.
3. Creating a project grants nothing over its contacts
The same identity created the project and, seconds later, was refused when
setting its essential contact. Project creation confers no authority over the
project's contact configuration; roles/essentialcontacts.admin is
separate.
4. An enforced constraint made a resource un-fixable by Terraform
Deleting Week 4's project should have been routine. The apply failed first, and the reason is worth sitting with.
During Week 4 I had manually added a block-project-ssh-keys
metadata key while testing the constraints. Terraform's configuration does not
contain it, so the apply tried to remove it — and
compute.managed.blockProjectSshKeys, enforced since Week 4,
refused the removal.
The configuration said one thing, the live resource said another, and organization policy forbade closing the gap. The instance was un-reconcilable with its own definition. Nothing was broken and nothing could be fixed; the only way forward was to destroy it, which was where this was going anyway.
5. The real limit on a project factory is a billing quota
Not Terraform. Not the module. Not IAM. A billing account caps how many
projects it will fund, this one caps at five, and the number appears in no
documentation I could find — it surfaces as
Cloud billing quota exceeded at the moment you exceed it.
A week about automating project creation could create exactly one project, and only after deleting one from the week before.
Security — Controls at Every Layer
The tag's permitted values are closed, and the module cannot extend them. It is handed a value ID. Nothing in the module's interface allows it to create a value, which is the difference between a tag that policies can rely on and a label that anyone can invent.
Every widening of CI is visible and separate. Four grants this week, in a configuration a human applies, each with its reasoning written next to it. Granting them where they were needed would have been shorter, would have worked, and would have made CI its own privilege administrator.
No default network, not negotiable. Google's default VPC is an auto-mode network in every region with permissive firewall rules nobody asked for. The module does not expose a way back to it.
An essential contact on every project. The organization
enforces essentialcontacts.managed.allowedContactDomains, inherited
from Google's security baseline, so an address outside the domain is refused at
apply rather than accepted and quietly ignored.
And the honest limitation: the constraint is in dry run. It refuses nothing today. It will be promoted after it has logged a real violation, which requires a project creation to evaluate — and the billing cap means the next one is not imminent.
Cost
$0. A project, a tag key with two values, an essential contact and a custom constraint. None of it bills. The project holds nothing that runs — no compute, no storage, no network of its own.
The interesting cost this week is not money. A project ID is
permanent and is never released for reuse.
katta698-gcp-dev-app-01 was spent in Week 4 and is gone forever,
even though the project was deleted. That is the resource this week actually
consumed, and it is the reason every ID in this lab carries an ordinal rather
than a description — a rebuild cannot have the old name back.
Cleanup
The tag key, its values and the constraint are meant to stay. They cost nothing and later weeks condition on them.
The project can go, and freeing its billing slot is the actual reason you
would bother. Note that the module sets deletion_policy = "PREVENT",
so tearing it down is a deliberate two-step: lift the policy, then destroy.
That is the guard working, and it is the same guard that made Week 4's teardown
a considered act rather than a fast one.
A tag key cannot be deleted while any value still exists, and a value cannot be deleted while anything is bound to it. Unwind in that order or the destroy fails halfway.
References
- Tags overview — tags as discrete IAM-controlled resources, referenceable from policy conditions.
- Labels overview — and the line that matters: labels cannot set conditions on policies.
- Scope organization policies with tags —
resource.matchTag, which is what the environment tag exists for. - Custom constraint supported services — lists
Projectat Preview, and stops there. - Project quota requests — a Console form, possibly with a payment.
Not answered anywhere I looked, and measured instead: which
fields a custom constraint can read on a Project; that
tagAdmin cannot bind a tag; that creating a project grants nothing
over its essential contacts; and the billing account's project cap.
Key Takeaways
- When an error names nothing, bisect the request. Six words cost an afternoon; making the request smaller until it succeeded cost twenty minutes and produced a table.
- Probe an unfamiliar resource before designing around it. "Supported" in a reference table does not tell you which fields are readable, and on a Preview resource nothing else will either.
- A factory's defaults are its product. Every toggle you add is a way for the thing to produce what it was meant to prevent.
- Creating and attaching are routinely different permissions. Twice in two weeks, in unrelated services. Expect it and read the role definitions rather than the error.
- The binding constraint is rarely the one you were working on. This week's limit was not Terraform, IAM or CEL. It was a number on a billing account that no code addresses.
Next week the hierarchy starts reporting on itself: what exists, what changed, and what drifted from the thing that was supposed to have created it.
Comments