📋 In This Post
Lab Overview
The problem. Week 1 built a resource hierarchy by running
terraform apply on a laptop. That works exactly once, for exactly
one person, and leaves no record of what was proposed before it happened. The
conventional fix is to give CI a service account key: a JSON
file holding a private key, pasted into a pipeline variable. It never expires,
it lives in a file, and it gets copied wherever is convenient.
Objective. Give the pipeline an identity with no stored credential at all — a workload identity pool trusting HCP Terraform's OIDC issuer, two service accounts split by run phase, and an attribute condition that decides who the issuer actually speaks for.
What reading the platform first changed. Listing the organization policy before writing any found seven constraints already enforced — Google's security baseline, applied automatically to every organization created on or after 3 May 2024. Two of them mean a key cannot be created and cannot be uploaded, so the planned opening move of the week was already done.
| Difficulty | Intermediate — the resources are few; the trust boundary is subtle and easy to leave wide open |
|---|---|
| Deployment | Terraform, hand-run on purpose and permanently — this is the layer that defines who CI is |
| Reproduce it | Lab guide — the attribute condition, the phase split, the denial |
| Cost | $0 — pools, providers, service accounts, bindings and token exchanges are all free |
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| Workload Identity Federation | Lets CI act as a service account with no key | Lets an identity from outside Google Cloud act as a service account, without a key. A pool holds external identities; a provider inside it names the issuer they come from. |
| Security Token Service | Performs the token exchange on every run | The endpoint that exchanges an external OIDC token for a short-lived Google federated token. Where the attribute condition is evaluated. |
| IAM service accounts | The two identities the pipeline assumes | Two of them, tf-plan and tf-apply, holding read and write roles respectively. |
| Organization Policy | Read, not written - it is why no key can exist | Not written this week — read. The baseline it already enforces is what makes a key impossible. |
| Project liens | Stops the seed project being deleted by anyone | Deletion refused at the API, whoever asks, until the lien is deliberately removed. |
| HCP Terraform | Runs the plans and applies, storing no credential | Runs the plans and applies, mints one OIDC token per run phase, and stores no GCP credential. |
The concept that reframes everything: the issuer is public.
https://app.terraform.io signs tokens for every HCP Terraform
customer in the world. Configuring it as a trusted issuer does not trust
your runs — it trusts all of them. The security control is
not the pool, the provider, or the issuer URL. It is the CEL expression that
inspects the token's claims and decides which of those customers you meant.
Impersonation is not authentication. The federated token is
not the service account. It is permission to ask to be one, and the
service account's own IAM policy decides whether the request is granted. That is
where the plan/apply split is enforced: each account grants
roles/iam.workloadIdentityUser to a principalSet scoped
to one value of the terraform_run_phase claim. Google refuses the
mismatch at exchange time, so it is a boundary rather than a convention.
Architecture and Workflow
No credential is stored on either side. Each run mints a token, exchanges it, uses it, and lets it expire.
The flow
- Read what the organization already enforces, before designing anything.
- Create the pool and a provider trusting
https://app.terraform.io, mapping five claims to attributes. - Write the attribute condition — the one piece that turns a public issuer into a specific trust relationship.
- Create
tf-planandtf-apply, and bind each to a phase-scopedprincipalSet. - Grant each identity the narrowest roles that work, individually rather than by basic role.
- Place a lien on the seed project, so deletion is refused at the API.
- Prove a plan really ran remotely, then attempt the shortcut and confirm it is refused.
The decision worth naming
This week is deliberately human-run, permanently. For
tf-apply to apply this configuration it would need
resourcemanager.organizations.setIamPolicy — which would let it
grant itself anything — and admin on the pool, which would let it rewrite
the attribute condition and trust whoever it liked. An identity must not
manage its own grants. So the lab has two layers: this week and the
bootstrap are hand-run and define who CI is; every other week runs
remotely as CI.
Prerequisites
- An organization with a seed project for bootstrap concerns — Week 1 supplies it. The pool, provider and both identities live there.
- An HCP Terraform organization and workspace. Note that the organization's real name may differ in case from every screen that displays it; see Troubleshooting.
roles/resourcemanager.organizationAdminon a human operator. This week cannot be run by CI, by design.- APIs:
iam,stsandcloudresourcemanager. - Terraform 1.9+ with
hashicorp/google~> 6.0.
Variables, role lists and commands are in the lab guide.
Implementation
Eighteen resources at deploy time, all free. The file carries thirty-four today, because later weeks added the roles CI needed — this is the layer where CI's permissions are defined, so it grows by design. Commands are in the lab guide.
Step 1 — Read the organization policy before writing any
Why: because what you are about to build may already exist.
The gcloud org-policies list call took two seconds and removed a
planned section of this week.
Checkpoint: seven constraints enforced, all sharing an
updateTime from the moment the organization was created.
Step 2 — The pool and the provider
Why: a pool holds external identities; a provider declares which issuer they come from and maps the token's claims to attributes that IAM can reason about.
attribute_mapping = {
"google.subject" = "assertion.sub"
"attribute.terraform_organization_name" = "assertion.terraform_organization_name"
"attribute.terraform_project_name" = "assertion.terraform_project_name"
"attribute.terraform_workspace_name" = "assertion.terraform_workspace_name"
"attribute.terraform_run_phase" = "assertion.terraform_run_phase"
}
Step 3 — The attribute condition, which is the actual control
Why: without a condition, the only thing between any HCP Terraform user and this organization is that they have not guessed the provider's resource name — which is not a secret and is printed in plan output.
attribute_condition = join(" && ", [
"assertion.terraform_organization_name == \"katta\"",
"assertion.terraform_project_name == \"GCP Platform Lab\"",
"assertion.terraform_workspace_name.startsWith(\"gcp-\")",
])
Three claims, narrowing in turn. The third is not decoration: other workspaces of mine live in the same HCP organization and carry tokens from the same issuer. The project and prefix checks are what keep them out of this organization.
Step 4 — Two identities, split by run phase
Why: HCP issues a token per phase, carrying a
terraform_run_phase claim, so the split can be bound in IAM rather
than trusted as convention.
resource "google_service_account_iam_member" "plan_federation" {
service_account_id = google_service_account.plan.name
role = "roles/iam.workloadIdentityUser"
member = "principalSet://iam.googleapis.com/${pool}/attribute.terraform_run_phase/plan"
}
Step 5 — Prove the plan ran remotely
Why: a workspace can be set to remote while the plan quietly executes locally against your own credentials, and the output looks identical. The tell is the first line — "Running plan in HCP Terraform".
tf-plan, refreshing Week 1's hierarchy through federation and
finishing with no changes.Step 6 — Attempt the shortcut
Action: ask for a service account key, as an identity that
holds iam.serviceAccountKeys.create.
Expected: a refusal that names the policy, and no key afterwards.
FAILED_PRECONDITION: Operation denied by org policy, naming
constraints/iam.managed.disableServiceAccountKeyCreation and
reason: CUSTOM_ORG_POLICY_VIOLATION. The following
keys list --managed-by=user returns nothing.Validation and Evidence
| Test | Expected | Actual | Status |
|---|---|---|---|
| Infrastructure deployed | 18 resources | 18 added, 0 changed, 0 destroyed | Pass |
| What the baseline already enforces | read before designing | 7 constraints, from organization creation | Finding |
| Workload identity pool | exists, correct issuer | https://app.terraform.io | Pass |
| Provider attribute condition | pins organization, project and workspace prefix | present, all three clauses | Pass |
| Service accounts | tf-plan and tf-apply, no keys | both; Key ID: No keys | Pass |
| Who may impersonate the apply account | the apply-phase principalSet only | correct | Pass |
| Plan identity cannot write | no write roles | five read roles, no write role | Pass |
| The plan actually ran remotely | "Running plan in HCP Terraform" | present; run visible in HCP as tf-plan | Pass |
| The shortcut, attempted | key creation refused, no key created | FAILED_PRECONDITION, CUSTOM_ORG_POLICY_VIOLATION; keys list empty | Pass |
The last row was unverified for two months. The figure proving the week's central claim was the one missing from the original capture. It was attempted and captured on 25 October 2026, as an identity holding the permission — the only way the result means anything.
It is the MANAGED generation doing the work. The legacy
constraint of the same name carries no policy and reads
enforce: false. Both generations have to be checked before
concluding anything about either — an absent policy reads identically to a
disabled one.
A test must not be able to create the thing it disproves.
The obvious way to write this check is to send the key to
/dev/null. That discards the key material — but if
the constraint ever failed to hold, the key would still exist server-side, on
the identity that can change the estate, created by the test meant to prove
none could exist. The failure branch has to clean up after itself, or the check
is a liability on exactly the day it finally fires.
Security and Cost
Security decisions
| Control | Decision and trade-off |
|---|---|
| No key, enforced not intended | Organization policy refuses creation and upload. Discipline is not a control; a denial from the API is. |
| The attribute condition is the trust boundary | A public issuer plus no condition trusts every customer of that issuer. Three claims narrow it to one project's workspaces. |
| Plan and apply are separate identities | Bound to different run phases, so a pull-request plan cannot reach a role that writes. Enforced by Google at exchange, not by convention. |
| Predefined roles, not basic | roles/viewer confers object read on every bucket. Replaced with five specific read roles — read-only is not the same as harmless. |
| Deletion protection at the API | A lien, not only prevent_destroy. Terraform's guard protects one configuration; the lien protects the resource. |
| Billing IAM deliberately out of scope | The apply identity spends against a billing account; it does not manage roles on it. That grant is made once, by a human, out of band. |
| Accepted gap | The pool trusts one HCP organization and a workspace prefix, not one workspace. Any workspace matching the prefix can obtain a token. |
Cost
$0. Workload identity pools, providers, service accounts, IAM bindings and liens are free, token exchanges are free, and HCP Terraform's free tier covers the runs. Which is the uncomfortable part: the keyless path costs nothing, takes an afternoon, and is still not what most tutorials teach.
Troubleshooting and Lessons
The given credential is rejected by the attribute condition
Problem: every token exchange refused, with an error naming no claim. Both ends looked correct.
Root cause: the HCP organization's real name is lowercase
katta. Every human-facing surface shows it capitalised — the
console URL, the cloud {} block, the workspace list — because
HCP resolves those case-insensitively. The OIDC token carries the true value, and
Google's CEL comparison is case-sensitive.
Lesson: when a comparison is case-sensitive on one side and not the other, the displayed value is not evidence. Cost roughly forty minutes.
A read-only role that could read every object in the estate
Problem: the plan identity was given roles/viewer
at the organization, on the reasoning that a plan only reads, so breadth costs
nothing.
Root cause: documented, and worth reading before granting.
roles/viewer confers
roles/storage.legacyObjectReader on buckets. So it was not "read the
hierarchy" — it was read every object in every project in the organization,
including projects that did not exist yet.
Resolution: five specific read roles.
Cleanup and Key Takeaways
Cleanup
None, deliberately. This is permanent infrastructure — every later week runs through it, and tearing it down would take the pipeline with it.
One thing resists cleanup on purpose: the lien on the seed project. Removing it
is a separate, explicit act requiring
resourcemanager.projects.updateLiens. That is the point of it.
Ongoing charges: none. Nothing here bills per hour.
Key takeaways
- Read the organization policy before you write any. A two-second command removed a planned section of this build. Organizations created on or after 3 May 2024 already refuse service account keys — and if yours predates that, nothing turned the constraints on for you.
- The issuer is public; the claims are the boundary. Trusting
app.terraform.iowithout an attribute condition trusts every HCP Terraform customer alive. That CEL expression is the security control, not the configuration around it. - A denial must be attempted by an identity that holds the
permission. An under-privileged attempt fails with a permission error
indistinguishable from the control working.
FAILED_PRECONDITIONrather thanPERMISSION_DENIEDis what makes the evidence mean something. - Read-only is not harmless, and a Terraform guard is not a
cloud guard.
roles/viewerreads object data;prevent_destroyprotects one configuration while a lien protects the resource. - Check where your plan actually ran. A remote workspace can still be planned locally, and the output looks the same. "Running plan in HCP Terraform" is the only proof the CI identity was involved at all.
What production would do differently
Condition on the workspace rather than a prefix, so one compromised workspace
cannot assume the apply identity. Scope the write roles below the organization
— serviceAccountAdmin at organization scope permits setting IAM
policy on any service account, including the apply identity itself, which
is a route to widening who may impersonate it. Use a variable set rather than
per-workspace variables. Separate the identity per environment. And alert on token
exchanges: a failed exchange is either a misconfiguration or somebody else trying,
and both are worth knowing about.
References
- Lab guide — the attribute condition, the phase split, the captured denial, known limitations
- Manage Google Cloud security baseline constraints
- Workload Identity Federation
- Find the right predefined roles
- Liens
- HCP Terraform — dynamic credentials with the GCP provider
- Source repository
Comments