Twelve policies, or twenty-six?

Why — The Problem This Solves

This week is the first one that runs entirely as the CI identity Week 2 built — no human credential anywhere, applying with a role that week granted in advance. That part worked. What did not work was the step before it: reading the organization I was about to change.

The instinct when you are handed an organization is to start hardening it. Pick the constraints that matter, switch them on, move on. That instinct assumes you can see the starting state, and on Google Cloud the obvious command does not show it to you.

There are two questions that sound identical and are not:

• What policy is set on this organization?
• What policy is enforced on this organization?

I asked the first, believed I had answered the second, and added a guardrail that had been active the entire time. Everything below follows from getting that wrong and then finding out.

What You Need to Know — Skills & Tools

Three properties of the Organization Policy Service drive every decision in this build. They are worth stating before any code, because each one has a failure mode that looks like success.

Policy inherits downward, and the closest policy wins. A policy written on a folder replaces the organization's for everything beneath it. It does not merge, and — this is the part that catches people — it does not have to be stricter. A folder can loosen what the organization set.

Therefore, writing a policy at the organization on a constraint that is already set replaces that spec. Hardening and loosening are the same API call with different contents. There is no "add to the existing policy" operation. If you codify an inherited constraint slightly wrong, it applies cleanly, the organization is left less protected than it was, and your state file asserts that everything is managed.

A dry-run policy is evaluated on every request and logged when it would have denied, while denying nothing. This is the mechanism that makes the rest safe: you can find out what a constraint breaks before it breaks anything.

One tooling note that is not optional. Everything here is written on the v2 Terraform resources — google_org_policy_policy and google_org_policy_custom_constraint. The older google_organization_policy cannot express a .managed. constraint at all. Writing this week against legacy constraint names would not be a stylistic preference; it would teach an API that is on the way out.

Architecture — How It Fits Together

This organization did not start empty. It inherited seven enforced constraints at creation, and reading them back showed all seven carrying the same updateTime to within a fraction of a second — one batch, applied by Google, not a set that accumulated over time.

So the shape of the week is not "turn on guardrails". It is: work out what the platform already decided, add only what it did not, and prove each addition is safe before it can deny anything.

What went in:

• compute.managed.requireOsLogin — VM access goes through IAM, not through keys pasted into instance metadata.
• compute.managed.blockProjectSshKeys — a project-wide key must not silently grant access to VMs that do not exist yet.
• compute.managed.disableSerialPortAccess — the serial console bypasses the network path entirely. No firewall rule, no VPC, no load balancer observes it.
• custom.requireTerraformLabelsOnBuckets — a custom constraint, covered below.

All four landed as dry run. One has since been promoted.

The exception, written down on purpose

Serial console access is refused across the organization, and explicitly allowed in workloads/dev. That is not a compromise; the thing the constraint defends against and the thing dev genuinely needs are the same thing — a way to see a machine too broken to reach over the network.

The reason to write it as a folder-level exception rather than leave it implicit is what happens otherwise. The first engineer with an unbootable dev VM either waits, or asks for the policy to be turned off. And it gets turned off at the organization, for everything, because that is where it was set. An exception at the narrowest node is what keeps the enforcement at every other node credible.

It is written as enforce = FALSE at the folder, not as a reset. A reset would return dev to whatever it inherits, which is enforcement. The override has to be explicit.

One custom constraint

Predefined constraints cover what Google anticipated. A custom constraint is how an organization's own rule becomes something the platform enforces, rather than something a reviewer has to notice.

The rule I picked was mine: every resource in this lab carries week, env and managed-by labels. That has been written in the repository's contributing instructions since Week 1, which means it has been enforced by precisely nothing. A label convention that lives only in a document is a convention that is already half broken and nobody has looked.

Buckets rather than VMs as the target, for a reason worth being honest about: this lab runs no compute, so a constraint on instances could not be tested without spending money to violate it. Bucket creation is free, so the denial is demonstrable — and a control you cannot demonstrate is a control you are taking on faith.

How We Built It — Step by Step

This is the first week in the lab that applies with no hand-run apply and no Google Cloud credential on the machine that starts it. HCP Terraform mints an OIDC token per run, Google's Security Token Service exchanges it, and the tf-apply identity built in Week 2 — which already holds roles/orgpolicy.policyAdmin at the organization — does the writing.

The order matters more than usual here:

1. Read the current policy first, with --effective. Not a formality. See the first challenge below.
2. Apply. Everything lands as dry_run_spec.
3. Read the violations out of the audit log.
4. Promote one constraint — exactly one — and apply again.
5. Validate against a real request, after a wait.

Step 4 is the method, not a suggestion. The enforcement switch in the configuration is a map keyed by constraint name rather than a single boolean, specifically so that constraints graduate one at a time. Flipping the set together works, and gives up the only thing dry run was for: knowing which constraint an audit entry belonged to.

What a dry-run violation actually looks like

Every article about dry-run mode says it logs violations. Almost none show you the record. Here is the one this deployment produced, from creating a bucket with no labels while the custom constraint was in dry run:

"metadata": {
  "@type": "type.googleapis.com/google.cloud.audit.OrgPolicyDryRunAuditMetadata",
  "constraint": "customConstraints/custom.requireTerraformLabelsOnBuckets",
  "dryRunResult": "DENIED",
  "liveResult": "ALLOWED",
  "resourceType": "storage.googleapis.com/Bucket"
},
"status": { "code": 7, "message": "POLICY_VIOLATED" }

The bucket was created. dryRunResult: DENIED beside liveResult: ALLOWED is the entire value of dry run in two fields: what the rule would have done, and what actually happened. Promoting this constraint stopped being a hopeful decision at that moment and became an informed one.

Two traps sit in that record, and both are the kind that cost an afternoon.

The status reads POLICY_VIOLATED with gRPC code 7 — PERMISSION_DENIED — on a request that succeeded. Anything alerting on status.code != 0, or grepping audit logs for POLICY_VIOLATED, will treat every dry-run evaluation as a live denial and report an outage that is not happening. That is a plausible way to lose confidence in a monitoring stack during exactly the week you need it.

And the entry is written to the project's audit log, not to an organization-level policy stream. The policy is set at the organization; the log is written where the request was served. Querying the organization for a policy log stream returns nothing and returns no error, which is indistinguishable from "there were no violations".

Challenges — What Actually Went Wrong

2. A clean plan is not evidence that an apply will work

terraform plan reported 7 to add, 0 to change, 0 to destroy. The apply failed immediately: Error 403 — SERVICE_DISABLED on orgpolicy.googleapis.com.

Organization policy is not stored in a project, so it is natural to assume no project needs anything enabled. But the write goes through the caller's quota project, and that project must have the Organization Policy API on. The read does not — the seven baseline constraints had listed perfectly, and the plan refreshed without complaint.

The dependency does not exist until the first write, so a plan can never surface it.

3. The API serializes policy changes, and the remote runner will not let you slow down

With the API enabled, the next apply failed differently: Error 409 — CONCURRENT_POLICY_CHANGES. The Organization Policy API serializes changes per parent. Terraform's default parallelism is 10, so it opened several SetPolicy calls against the same organization at once.

The failure is partial and non-deterministic — the first attempt landed two of the four policies before failing, which is worse than a clean failure because re-running appears to make progress.

The documented answer is -parallelism=1. It does not exist here:

Error: Custom parallelism values are currently not supported

HCP Terraform does not support setting a custom parallelism value at
this time.

This is a consequence of the week running remotely rather than anything about organization policy, and it forced the better fix. On local execution the answer is a flag — which leaves the configuration wrong and moves the correction into an operator's memory, where it works for exactly as long as the person who learned it is the person running it. With no flag available, the ordering has to be an edge in the dependency graph: each policy resource depends on the one before it. That is a property of the code, and it survives being run by someone who was never told.

The map-driven for_each version was shorter and read better. It also did not work.

dry run  -> enforced    75s   99s   75s
enforced -> dry run    108s  103s   77s

Here is the refusal itself, which is the single piece of evidence the rest of this week rests on:

$ gcloud storage buckets create gs://wk03-denytest-... --uniform-bucket-level-access

ERROR: HTTPError 412: orgpolicy:projects/_/buckets/wk03-denytest-...
violates customConstraints/custom.requireTerraformLabelsOnBuckets.

Details: Storage buckets must carry the three labels this lab requires of
every resource. managed-by must be terraform, so a bucket created by hand
in the console is refused.

- '@type': type.googleapis.com/google.rpc.PreconditionFailure

412 Precondition Failed, not 403 — and the description field written in the Terraform is quoted straight back to whoever hits it. That is the argument for writing that field for the person who will be stopped by it, rather than as a change-log entry nobody reads.

Security — Controls at Every Layer

Nothing here touches the inherited baseline. The obvious move on reading seven enforced constraints is to bring them into Terraform so the whole policy surface is in code. It is the wrong move. Writing them into the configuration does not adopt them — it issues SetPolicy with whatever the file says, replacing the inherited spec. Two of the seven carry parameters read out of the live organization, including a customer ID, so codifying them means either committing account identifiers to a public repository or plumbing them through variables to reproduce a spec Google already applies correctly. The failure mode is quiet: a transcription slip does not error, it applies.

The denial was tested as an identity that holds the permission. This matters more than it sounds. An attempt made by an under-privileged identity fails with a permission error that looks identical to the constraint working. A test that asserts only on "the request failed" would pass against a completely broken control.

The constraint's description is written for the person who hits it. When the denial lands, Google quotes that field back to the caller verbatim. It is not a change-log entry; it is the error message.

Reading audit logs required a role that organization administrator does not include. roles/logging.viewer was granted out of band, by a human, and is not managed by this configuration — the same separation Week 2 applied to the billing grant. An identity should not manage the grants that govern it.

Cost

Zero. Organization policy is metadata on the resource hierarchy, not infrastructure. Nothing in this week bills per hour, per request, or per gigabyte.

Two APIs are enabled on the seed project — Organization Policy and Cloud Logging. Enabled APIs cost nothing on their own, and the log volume this generates sits inside the free tier.

The buckets created to test the constraint existed for seconds and were deleted immediately, including by the validation script, which cleans up whether the request succeeded or failed.

Cleanup

There is nothing here worth tearing down to save money, so the teardown exists for correctness rather than cost. scripts/cleanup.sh destroys the policies and the custom constraint, and then states plainly what it does not undo: the two enabled APIs, the out-of-band logging role, and the Google security baseline, which was never managed here and should not be disturbed.

One question remains genuinely open, and I would rather say so than assert it. Google's documentation contradicts itself on whether a deleted custom constraint's name can be reused — one page says never, another says within minutes. Which is true decides whether that script can honestly claim a clean teardown. It is scheduled as a measurement rather than quoted from either page, and it has not been run yet.

References

Every source that settled something, and the questions none of them answered — the second list is the more useful one.

• Managed constraints — Compute Engine: confirmed the compute.managed.* names are current and replace the legacy ones.
• Restrict IAM service account usage: the line about the Gemini API being allowed by default, which I read before deploying and did not act on hard enough.
• Dry-run policies.
• Custom constraints.
• Understanding hierarchy evaluation: that the closest policy wins, which is what makes the folder exception work and what makes an organization-level write replace rather than merge.
• terraform-provider-google #21401: why this week is written on the v2 resources.

Not answered anywhere I looked, and measured instead: that gcloud org-policies list omits default-enforced managed constraints; where dry-run violations are actually logged; the parameter schemas of the parameterised managed constraints; and the delay between writing a policy and it being enforced.

Key Takeaways

• Read the effective policy, not the set policy. If you take one thing from this week, take --effective. The default command answers a question you were not asking.
• When two of your own tools disagree about a number, that is the finding. Twenty-six against twelve. I saw both figures on the same screen, read past the gap, and shipped a guardrail that was already live.
• Dry run turns a guess into a decision. Two fields — dryRunResult and liveResult — are the difference between enforcing a constraint because you hope it is safe and enforcing it because you watched what it would have blocked.
• Write the exception down. An undocumented exception becomes a request to disable the policy at the level it was set, which is the organization.
• A constraint you cannot demonstrate is one you are taking on faith. Pick a target you can afford to violate.
• Constraints that force better code are a gift. Losing -parallelism produced a configuration that is correct for whoever runs it next, rather than one that depends on remembering a flag.
• One observation is not a measurement. The enforcement lag in this post started as a single success followed by a single failure, written up with the same confidence as everything reproducible around it. Repeating it three times turned a vague "about two minutes" into 75 to 100 seconds, and turned up the reverse lag — which is the more useful number — entirely by accident.

Next week the hierarchy gets something to hold. Everything so far has been control plane — who may act, and what the platform refuses. That is the right order, and it is nearly finished.