Business Challenge
Post #20 ended by saying no custom constraint should reach production without being watched first. This is how that is done, and the four ways it goes wrong.
Dry-run mode is not available for every constraint. It works with custom constraints, with managed constraints, and with four named legacy managed constraints — restrict service usage, restrict endpoint usage, restrict TLS versions and restrict TLS cipher suites. Google is blunt about the rest: attempting to create an organization policy in dry-run mode using any other constraint results in an error.
Correct approachCheck the constraint's kind before planning a rehearsal. If it is a legacy managed constraint outside those four, there is no dry run, and the rollout plan has to be built on staged scope instead — one folder at a time rather than one mode at a time.
A dry-run policy only sees operations that actually happen while it is running. It logs what would have been denied, which says nothing about a quarterly batch job, a disaster-recovery rebuild, or the Terraform apply that only runs when someone onboards a new team.
Correct approachRun it long enough to cover the estate's real cycle, not the sprint. And treat "no violations" as evidence about the window observed, not a property of the policy.
Nowhere you will stumble across. Organization policy audit logs are generated based on whether the operation is allowed or denied by the policies enforced on a given resource, and finding the dry-run cases means asking for exactly the combination where the two verdicts differ.
Correct approach
Query for it directly: protoPayload.metadata.dryRunResult = "DENIED" AND protoPayload.metadata.liveResult = "ALLOWED". That pair — the dry run refused it, the live policy let it through — is the definition of a would-be violation.
A dry run cannot tell you. It observes operations, and an existing non-compliant resource performs no operation by sitting there. This is #19's problem exactly: enforcement is not retroactive, so a resource already in the forbidden state is in violation and keeps running.
Correct approachThat is the other tool's job. Policy Simulator previews the impact of a proposed policy and returns the resources that violate it — the standing population, not the traffic.
Architecture
There are two rehearsal tools and they answer different questions. Choosing the wrong one produces a confident, clean result about something you were not asking.
The two questions
| Policy Simulator | Dry-run mode | |
|---|---|---|
| Question it answers | What already violates this policy? | What would this policy refuse from now on? |
| Looks at | Existing resources, as they stand. | Operations, as they happen. |
| When | Before anything is set. It previews the impact of a new constraint or policy before it is enforced. | While it is set. It is created and enforced similarly to other policies. |
| Blocks anything | No — nothing is set. | No — it does not block any operations when enforced. |
| Output | A list of resources that violate the proposed policy. | Audit log entries where the dry-run and live verdicts differ. |
| Constraint support | Custom and managed. It cannot test changes to legacy managed constraints. | Custom, managed, and four named legacy managed constraints. |
The pairing is not redundant. The simulator finds the resources that will have to be fixed before enforcement can succeed — the remediation backlog #19 warned would otherwise be invisible. The dry run finds the workflows that will break after it, which no inventory of existing resources can predict because they have not happened yet.
A policy can hold both specs at once
The mechanism is one extra field. A dry-run policy uses dryRunSpec where a live one uses spec, and both may sit in the same document:
name: RESOURCE_TYPE/RESOURCE_ID/policies/CONSTRAINT_NAME
dryRunSpec:
rules:
- enforce: true
You can set a live organization policy and a dry-run organization policy in the same YAML file by defining both spec and dryRunSpec. That is what makes a tightening safe to stage: keep the current live rule enforcing, add the stricter rule as a dry run, and compare the two verdicts on real traffic before promoting one to the other.
Because the two specs are separate fields, they are removed separately. gcloud org-policies set-policy with --update-mask=dryRunSpec clears the dry run and leaves the live policy alone, while gcloud org-policies delete removes both. During an incident, the difference between those two commands is the difference between ending an experiment and dropping a guardrail.
The query that turns logs into an answer
A dry run produces nothing you will notice by accident. Audit logs are generated based on whether the operation is allowed or denied by the policies enforced on a resource, so both verdicts are recorded on the same entry and the interesting case is where they disagree:
protoPayload.metadata.dryRunResult = "DENIED"
AND protoPayload.metadata.liveResult = "ALLOWED"Read that as the report: the proposed rule would have refused this, and today's rules permitted it. Every row is a workflow that breaks on the day you promote the dry run, with the principal, the resource and the timestamp attached — which is precisely the list needed to warn the right team before rather than after.
Both tools exclude legacy managed constraints — the simulator entirely, dry-run mode except for four. Those are the oldest constraints, the ones with the widest adoption and the most existing documentation, and they are the ones a team reaches for first. So the constraints easiest to adopt are the hardest to rehearse, and the ones that support rehearsal are the ones requiring more work to write. That is worth factoring into the choice made in #20: a managed constraint is preferable to its legacy equivalent partly because it can be tested at all.
Why This Architecture Holds Up
How long is long enough
The honest answer is one full cycle of whatever the estate does, and that is rarely a week. A dry run observes operations, so its coverage is the coverage of your traffic:
- Daily deploys are covered within days, and give false confidence quickly.
- Monthly batch and reporting jobs need a month, or a deliberate manual run inside the window.
- Quarterly and annual work — DR exercises, certificate rotations, capacity rebuilds — will not appear at all. Trigger them on purpose or accept that they are untested.
- Onboarding paths only run when somebody onboards. If none happened, that path is unobserved.
Add the propagation delay from #19: changes take up to 15 minutes to be fully enforced, so a dry run started and queried in the same ten minutes is measuring nothing.
The order that works
- Simulate, to get the list of existing resources that violate the proposal. Fix or exempt them; this is the remediation project, and it is usually the long pole.
- Set the dry run alongside whatever live policy exists, at the scope you intend to enforce at — a dry run set on a folder inherits to its descendants like any other policy, so scope it where the real one will sit.
- Query the audit log for the verdict mismatch, on a schedule rather than once, and route the results to the teams that own the workloads.
- Promote by moving the rule from
dryRunSpectospeconce the mismatches stop, or once every remaining one is understood and accepted.
For a legacy managed constraint outside the supported four, the only rehearsal available is a smaller blast radius. Set the real policy on one low-risk folder, watch what breaks, then move it up the hierarchy. It is slower and it does deny things — but it denies them somewhere chosen, which is the entire property a dry run was going to buy. Setting an untestable constraint straight at the organization node is the version of this with no rehearsal at all.
Who can do it, and why that matters here
Setting a dry-run policy needs the same role as setting a live one: Organization policy administrator, roles/orgpolicy.policyAdmin, on the organization. There is no lesser permission for the safe version.
That is defensible — a dry run is still a policy object on a production resource, and the field distinguishing it from an enforcing one is a single word. It also means the rehearsal cannot be delegated to the team that owns the workload, which is an argument for routing the audit-log results to them rather than handing over the role.
Key Architecture Decisions
| Decision | Choose this | Because |
|---|---|---|
| Finding existing violations | Policy Simulator | It returns the resources that violate a proposed policy; a dry run cannot see a resource that performs no operation. |
| Finding workflows that will break | Dry-run mode | It logs what would have been denied, on real traffic, without denying it. |
| Before planning either | Check the constraint's kind | Any constraint outside custom, managed and the four named legacy ones errors on dry run. |
| Choosing between a legacy and managed constraint | Managed, partly for testability | The simulator cannot test legacy managed constraints at all. |
| Tightening an existing policy | Both specs in one file | spec keeps enforcing while dryRunSpec measures the stricter rule. |
| Reading the results | Query for the verdict mismatch | dryRunResult = "DENIED" with liveResult = "ALLOWED" is the definition of a would-be violation. |
| How long to run it | One full business cycle, not one sprint | A dry run only observes operations that occur while it is set. |
| Interpreting a clean dry run | As evidence about the window, not the policy | Quarterly and annual workflows will not have run. |
| Where to set the dry run | The scope the live policy will occupy | It inherits to descendants like any other policy, so a narrower scope rehearses less. |
| Ending an experiment | --update-mask=dryRunSpec, not delete |
delete removes the live policy too. |
| Untestable constraints | Stage the scope, folder by folder | A chosen blast radius is the only rehearsal left when dry run is unavailable. |
| Delegating the rehearsal | Route the logs, not the role | Dry run needs policyAdmin on the organization, same as enforcing. |
Closing Thought
Almost everything else in an architecture can be tried somewhere cheap first. There is a test cluster, a staging project, a canary. Organization Policy has none of that by construction — the whole point of a control that ignores who is asking is that it sits above the environments, and you cannot have a staging copy of the thing that constrains staging.
Dry-run mode and Policy Simulator are the compensation, and they are good ones: between them they answer what already fails and what would fail next, which is most of what a staging environment is for. The uncomfortable part is where they stop. The constraints a team adopts on day one, from a blog post or a benchmark, are largely legacy managed constraints — and those are the ones neither tool will rehearse. Knowing that in advance turns it from a surprise into a choice: prefer the constraint you can test, and where you cannot, spend the caution on scope instead.
#22 gets concrete: the organization policies worth setting on day one — which handful actually earn their place in a new estate, what each prevents, and which of them you will be able to rehearse first.
Comments