Business Challenge
A platform team automates project vending. The pipeline does four things in order: create the project, attach it to the billing account, grant the deploying service account a role on it, then run the first deployment as that service account.
Always at the fourth step, always with a permission error naming a role the console plainly shows as granted. A manual run has a human-sized pause between the steps; CI does not.
Correct approachRead that pause as part of the test. A sequence that only passes while somebody is typing between the steps is depending on timing rather than on the grant.
create, delete, move, patch and undelete return a long-running Operation rather than the resource. Code that reads the response as the created project works whenever the operation happens to finish quickly.
Correct approachPoll the Operation until it reports done. The response is a handle, not a result.
An allow policy change takes typically 2 minutes to propagate and potentially 7 minutes or longer. The API and the enforcement path were both being truthful about different things.
Correct approachWait on the permission rather than on the API acknowledgement, and split granting from using into stages that tolerate a gap between them.
The second attempt succeeded having changed nothing at all. A transient error implies something failed and then worked; here nothing failed — the retry simply happened after the window had closed.
Correct approachTreat a retry that changes nothing as evidence of a propagation window rather than of flakiness. The same delay reappears anywhere a design assumes an access change is immediate.
It was not transient and it was not a race in their code. It is the documented behaviour of the platform, and the team had built a pipeline on an assumption the platform never made: that when a write returns, the thing it wrote is in effect everywhere.
A write to Resource Manager returns an Operation — a handle you poll until it reports done. That is the first wait, and it is at least obvious, because the response is clearly not the resource you asked for. The second wait is invisible: once the operation has completed, the change still has to reach the systems that enforce it. An allow policy change takes typically 2 minutes and potentially 7 minutes or longer. Nothing in the API response tells you that, and nothing in the console shows it.
Architecture
Resource Manager is the API for the hierarchy itself — organizations, folders, projects, tag keys and values, and the liens that protect them. Every other service in this series sits inside a container this API creates and governs, which is why it comes before any of them.
Half the methods are asynchronous, and it is not the half people expect
The projects resource supports create, delete, get, getIamPolicy, list, move, patch, search, setIamPolicy, testIamPermissions and undelete.
Of those, create, delete, move, patch and undelete return a long-running Operation. The reads — get, list, search — are ordinary synchronous calls that return what you asked for.
The consequence is that every mutation of the hierarchy is a two-step conversation. You do not create a project; you start a project creation and receive a handle. Code that treats the response as the created resource is reading the wrong object, and it will appear to work whenever the operation happens to complete quickly.
A project has two names and one of them is the real one
The canonical resource name is projects/PROJECT_NUMBER — the generated number, not the ID you chose. API calls accept either, which is convenient and is also why the distinction goes unnoticed until something depends on it.
Prefer the number in anything durable. The ID is the human handle and the one in URLs; the number is what the platform means. When you eventually read an audit log or an IAM binding that Google generated rather than you, it is the number you will see.
The lifecycle has a state you can come back from
A project in ACTIVE is in normal operation. Deleting it moves it to DELETE_REQUESTED, which is generally reversible by invoking projects.undelete — the API-level view of the soft delete described in post #1, where the project is fully removed after 30 days and counts against quota throughout.
Treat "generally reversible" as the careful phrase it is. The project record comes back; the resources that lived inside it are a separate question, and some of them will not.
Liens: the brake that is not a permission
A lien placed on a project blocks the project's deletion until the lien is removed. It is not an IAM condition and not an organization policy — it is a separate object attached to the project whose only job is to refuse one operation.
What makes it useful as a design element is who can lift it. Removing a lien requires the Project lien modifier role (roles/resourcemanager.lienModifier), carrying resourcemanager.projects.updateLiens. That is a different grant from project ownership, so an owner who can delete the project cannot quietly clear the thing stopping them. Two people, or at least two deliberate steps.
Services must be enabled before they exist for you
To use most Google Cloud APIs and services you must first enable them in a project. Enabling associates the service with the project and enables billing for it where billing is enabled on the project.
For automation this is another ordering constraint hiding in plain sight: a freshly created project has almost nothing switched on, so a pipeline that creates a project and immediately calls a service API is making a call against something not yet present in that project.
Why This Architecture Holds Up
The failure in the opening story has a precise shape, and it is worth stating exactly, because the fix follows from it.
The pipeline polled correctly. The operation genuinely completed. The grant genuinely existed — it was visible in the console and in getIamPolicy immediately. What had not happened was propagation: an allow policy change takes typically 2 minutes and potentially 7 minutes or longer before every system that enforces it agrees.
So there is a window, minutes wide, in which the API will tell you a principal has a role and a service will tell you it does not. Both are being truthful about different things. The control plane has recorded the change; the enforcement path has not finished hearing about it.
A retry a few minutes later succeeds having changed nothing at all, which is exactly what makes this so easy to misdiagnose as flakiness. A transient error implies something failed and then worked. Here nothing failed — the second attempt simply happened after the window closed. Adding a retry does work, but understanding why matters, because the same delay will reappear anywhere a design assumes an access change is immediate.
Groups are slower, and most designs grant through groups
The advice everywhere, including in this series, is to grant roles to groups rather than to individuals. That advice is right, and it has a cost worth knowing: a group membership change takes typically several minutes and potentially hours or longer to propagate — substantially slower than a direct policy change.
A design that grants through groups inherits the slower of the two. For humans that is invisible and irrelevant; nobody notices that their new access took four minutes. For automated joiner-mover-leaver flows, and for any break-glass procedure that assumes adding someone to a group grants access immediately, it is a real property to design around. A break-glass path that routes through group membership is a break-glass path with an unpredictable delay in front of it.
What this means for how you write automation
The general rule: separate the act of granting from the act of using. Anything that provisions access and then immediately exercises it is depending on a timing property the platform explicitly does not offer.
In practice that means waiting on the thing you actually care about rather than on the API's acknowledgement — poll until the permission works rather than until the write returns — and building pipelines in stages that can tolerate a gap between them, rather than one long transaction that assumes each step is fully effective the moment the previous one returns.
Key Architecture Decisions
| Decision | Choose this | Because |
|---|---|---|
| Handling a create, delete, move, patch or undelete | Poll the returned Operation until done | These five are asynchronous. The response is a handle, not the resource, and code that reads it as the resource works only when the operation happens to be fast. |
| Granting access then using it in the same pipeline | Separate the stages, and wait on the permission rather than on the API | An allow policy change is typically 2 minutes and potentially 7 or more. The operation completing tells you nothing about enforcement having caught up. |
| Granting to groups or to principals directly | Groups, with the delay designed for | Groups remain right for manageability, but membership changes take several minutes and potentially hours — slower than a direct policy change, and the slower path is the one you inherit. |
| Break-glass access | Do not route it through group membership alone | An emergency path whose activation takes an unpredictable number of minutes is not an emergency path. Grant directly for break-glass, or pre-provision it. |
| Referring to a project in durable systems | The project number | It is the canonical resource name. The ID is the human handle; the number is what generated bindings and audit records will show you. |
| Protecting a project that must not be deleted | A lien, not just careful IAM | It blocks deletion outright, and lifting it needs roles/resourcemanager.lienModifier — a different grant from ownership, so the owner cannot quietly clear their own obstacle. |
| A newly created project | Enable services explicitly, as a step | Most APIs must be enabled before use. A pipeline that creates a project and immediately calls a service API is calling something not yet present there. |
Find every place that changes access and then uses that access. If the gap between the two is measured in seconds, it is relying on a guarantee that does not exist — and it will pass in testing, because a single manual run has a human-sized pause in the middle of it. That pause is the bug, hiding.
Closing Thought
Resource Manager is the least visible layer in this series and the one most likely to produce a failure nobody can reproduce. Its rules are not about any particular service: the same asynchrony and the same propagation delay apply whether the project holds one bucket or a thousand VMs.
The single idea worth carrying forward is that Google Cloud is honest about being eventually consistent, and publishes the numbers — 2 minutes typically, 7 or more sometimes, hours for group membership. Those are not caveats buried in a footnote; they are the operating characteristics of the control plane. A platform built as though they were zero will work in demonstrations and fail intermittently in production, which is the most expensive way for a design to be wrong.
Post #4 goes one level down, to the services themselves.
#4 covers enabling services: why an API has to be turned on before it exists in a project, what enabling actually changes, what it means for billing, and why the list of enabled services is a governance surface rather than a checkbox.
Comments