The rotation that rotates nothing

Lab Overview

The problem. A database password sits in Secret Manager. The console offers a rotation period, you set it to a sensible interval, and the page then tells you when the next rotation is due. Everything about that screen suggests the credential will change on its own. It will not. Eighteen months later every service is still reading the password somebody typed in on the first day, and nothing anywhere has reported a problem — because nothing is wrong, exactly. The schedule is running. It is just not doing what the word implies.

The objective. Deploy a secret encrypted with a key this lab owns, give it the shortest rotation period Google allows, and measure what actually changes when that schedule fires. Then build the piece Google does not ship, fire the schedule again, and measure the difference.

Deployment method. Terraform, run on HCP Terraform agents with no stored credentials — the workspace mints an OIDC token per run and exchanges it for a short-lived Google token through Workload Identity Federation, with separate identities for the plan and apply phases. That machinery was built in Week 02 and every week since rides on it.

Cloud Services and Concepts

What a customer-managed key actually gives you

By default Google encrypts everything in Secret Manager with keys Google owns and manages. CMEK replaces that with a key you create in Cloud KMS. The encryption is identical; what changes is who controls the key's lifecycle.

The part worth internalising is who performs the cryptography. It is not the caller. Secret Manager decrypts as its own service agent — a Google-managed identity of the form service-<project-number>@gcp-sa-secretmanager.iam.gserviceaccount.com — and that agent needs an explicit grant on the key. Owning the key confers nothing; creating the secret confers nothing. If the agent does not hold roles/cloudkms.cryptoKeyEncrypterDecrypter on that specific key, the secret cannot be created at all.

There is also a location constraint that reads like a preference and is not one. A secret using automatic replication can only be encrypted with a key in the global multi-region. A regional key forces user-managed replication, with one key per replica location.

Rotation, twice, meaning two different things

Both services in this lab offer something called a rotation period. They are configured within forty lines of each other in the same Terraform file. They do not do the same kind of thing.

Google's documentation is not evasive about this. It says the service “triggers a SECRET_ROTATE message to the designated Pub/Sub topics” and that you “must configure a Pub/Sub subscriber to receive and act on” it. The gap is not between the docs and reality — it is between the docs and the word.

Architecture and Workflow

1. A Cloud KMS key ring and key are created in the global multi-region, with automatic rotation every 90 days.
2. The Secret Manager service agent is created explicitly and granted encrypt/decrypt on that key — before anything tries to use it.
3. A Pub/Sub topic is created, and the same service agent is granted publisher on it.
4. The secret is created with automatic replication, CMEK pointing at the key, the topic attached, and a one-hour rotation period.
5. The schedule fires. A SECRET_ROTATE message lands on the topic. Nothing else happens.
6. A Cloud Run function subscribed to the topic adds a new secret version and disables the previous one.
7. The schedule fires again. This time the version count moves.

Design decisions worth stating

The deployment is staged, and the staging is the experiment. A deploy_rotator variable gates every stage-two resource. This is not a convenience flag. Proving that a schedule rotates nothing requires one notification to fire with nothing subscribed, so the version count can be shown unchanged either side of it. Once the rotator exists there is no route back to that observation without rebuilding the week.

Nothing consumes the secret. A workload reading it would add a second thing to debug and would prove nothing extra about the control.

Prerequisites

• A folder hierarchy with a dedicated security project (Week 01). This week is the first to use the cloudkms and secretmanager APIs that week enabled there three months ago.
• Keyless CI through Workload Identity Federation (Week 02), with separate plan and apply identities.
• Project-scoped IAM for the pipeline — cloudkms.admin, secretmanager.admin, pubsub.admin and serviceusage.serviceUsageAdmin on the security project, plus viewer equivalents for the plan identity. Granted on one project rather than the platform folder, because a folder grant would also reach the network hub and logging projects, neither of which has any business holding a key.

Enabling an API is not holding a permission. Week 01 enabled Cloud KMS on this project in August; the pipeline still could not create a key ring in November until it was granted one.

Implementation

What was deployed

Step 1 — The key, and the agent that uses it

Why: the secret cannot be created until the key exists and the service agent can use it.

resource "google_kms_crypto_key" "app_db_password" {
  name     = "wk12-app-db-password"
  key_ring = google_kms_key_ring.secrets.id
  purpose  = "ENCRYPT_DECRYPT"

  version_template {
    algorithm        = "GOOGLE_SYMMETRIC_ENCRYPTION"
    protection_level = "SOFTWARE"
  }

  rotation_period            = "7776000s"   # 90 days — genuinely rotates
  destroy_scheduled_duration = "86400s"     # NOT the 30-day default
}

Validation: protectionLevel reads SOFTWARE and destroyScheduledDuration reads 86400s. The second one is a cost control, and the reason is in the Cost section.

Step 2 — The secret, after the topic

Why the order: Secret Manager validates at creation time that it can publish to every configured topic, so a missing publisher binding surfaces not as a broken rotation later but as a secret that will not exist.

replication {
  auto {
    customer_managed_encryption {
      kms_key_name = google_kms_crypto_key.app_db_password.id
    }
  }
}

rotation {
  rotation_period    = "3600s"              # Google's minimum
  next_rotation_time = var.next_rotation_time
}

topics {
  name = google_pubsub_topic.rotation_events.id
}

Validation: the console shows the key, the schedule and the topic on one page.

Step 4 — Fire the schedule with nothing subscribed

Why: this is the measurement the post rests on, and it is perishable.

measure-rotation.py records the version count and newest enabled version, waits for the SECRET_ROTATE message to arrive on an observer subscription, waits a further 60 seconds so that “unchanged” cannot mean “too early”, and reads the count again.

The independent confirmation matters as much as the measurement: nextRotationTime advanced from 05:06:19Z to 06:06:19Z, exactly one rotation period. Secret Manager only reschedules after the event, so the schedule firing is not in doubt. What it changed is.

Step 5 — Write the rotator, and fire it again

The function keys on the event type, and that is not defensive clutter:

if event_type != "SECRET_ROTATE":
    print(f"ignoring {event_type!r}: only SECRET_ROTATE triggers a rotation")
    return

Adding a version publishes SECRET_VERSION_ADD to the same topic, and disabling one publishes SECRET_VERSION_DISABLE. A rotator that acted on every message would rotate because it had just rotated, forever.

Validation and Evidence

What the denial proves, and why its error code matters. Disabling the key version touches neither the secret nor a single IAM binding. The caller still holds secretmanager.versions.access. The refusal comes back as FAILED_PRECONDITION on the Cloud KMS resource, not PERMISSION_DENIED — so nothing about who is asking changed. Only whether Secret Manager can still decrypt.

The spread is the finding, not the number. Three trials gave 70s, 2s and 27s. A disable reaches Secret Manager somewhere between near-instantly and over a minute, so any fixed wait shorter than roughly 70 seconds will sometimes report a working control as broken. The first version of this test slept 30 seconds, read once, saw it succeed, and concluded the control did not exist.

Security and Cost

Security

• The rotator cannot read the secret it rotates. It holds secretVersionAdder and secretVersionManager on one secret, and deliberately not secretAccessor. A rotator writes a new value; it never needs the old one. The identity handling this credential every hour cannot read it.
• Building and rotating are separate identities. The build identity holds cloudbuild.builds.builder and no secret permission; the rotator holds no build permission.
• Neither pipeline identity can read the value. Terraform manages the secret and never consumes it. secretmanager.viewer reads metadata — labels, replication, schedule, version list — and not the payload.
• No service account keys anywhere, as in every week of this lab.
• The widest grant, named rather than buried: cloudkms.admin carries cryptoKeys.setIamPolicy, so the pipeline decides who may encrypt and decrypt with any key in this project. Unavoidable for a pipeline that creates CMEK keys — the service agent's grant has to be written by something — and it does not carry the cryptographic operations themselves.

Cost

This is the first week of this lab whose main resource has no free tier at all. Not because it is expensive — six cents a month is not expensive — but because every previous $0 was a free-tier $0, and this one bills from the first hour.

Prices below are read from this lab's own billing pricing export, built in Week 06, rather than from a rate card.

The teardown does not stop the key charge, and that is a finding rather than an oversight. Google counts a key version as active — and billable — if it is Enabled, Disabled, or Scheduled for destruction. Only DESTROYED stops it, and a destroyed version first spends destroyScheduledDuration in DESTROY_SCHEDULED, which defaults to 30 days. This lab sets it to 24 hours precisely so the week does not meter for a month after its cleanup reports success.

Two consequences follow that are easy to get backwards. Disabling a key stops it working and does not stop it charging. And leaving the hourly rotation running accumulates secret versions — this deployment reached nine within a day, past the six-replica free tier.

Measured cost: not yet available. The charges have not posted at the time of writing. This section will be updated with the observed figure rather than left as an estimate.

Troubleshooting and Lessons

Every API bills the caller unless told otherwise

Problem: the first apply failed with “Cloud Key Management Service (KMS) API has not been used in project <number>” — naming the seed project, where the pipeline's identity lives, not the security project where the key was being created. Cloud KMS was enabled on the security project the whole time.
Root cause: without an explicit quota project the provider sends no X-Goog-User-Project header, so Google attributes the call to the project owning the credentials.
Resolution: user_project_override = true and billing_project pointing at the security project — rather than enabling crypto APIs on the seed project, which would quietly make it somewhere keys could live.
Lesson: earlier weeks recorded this as “some APIs bill the caller.” That framing is too narrow. Every API bills the caller unless told otherwise; most of the time the caller and the resource happen to be the same project, which hides it.

Two of this lab's own guardrails blocked the rotator

Problem: deploying the function failed twice, both times on controls built earlier in this series.
Root cause: Week 03's custom constraint requireTerraformLabelsOnBuckets refused the gcf-v2-sources bucket that Cloud Functions creates for itself. Then Week 09's disabled default Compute Engine service account turned out to be the identity Cloud Build builds as.
Resolution: satisfy both rather than weaken either. The staging bucket name is deterministic, so it is pre-created with the required labels; and the build runs under a dedicated identity instead of the dormant account Week 09 deliberately retired.
Lesson: a guardrail written to stop a careless human will eventually block a Google service doing something reasonable. That is the control working, and the cost is real. Carving an exception is the easy answer and usually the wrong one.

A misconfigured Eventarc trigger looks like silence

Problem: the rotator deployed ACTIVE, the schedule fired, and no version appeared.
Root cause: eventarc.eventReceiver is not enough. A second-generation function is a Cloud Run service, so delivery is an authenticated HTTP call needing run.invoker on the service.
Resolution: grant it on the service, not the project.
Lesson: because the trigger retries, this never appeared as a failure — just silence at the secret and a retry storm in the logs. When a notification-driven thing does nothing, read the consumer's logs before suspecting the producer.

The API and the provider do not use the same words

Terraform's schema calls the replication block auto; the API returns it as automatic. A validation script written in the provider's vocabulary and run against gcloud read an empty string and reported a correctly-configured secret as broken. The point of validating against the cloud rather than Terraform state is to be independent of the configuration — borrowing the configuration's field names gives that up silently.

Cleanup and Key Takeaways

Cleanup

terraform destroy removes the secret, the topic, the subscription, the rotator and its identities. It does not remove the key, and the provider's own documentation says so plainly:

CryptoKeys cannot be deleted from Google Cloud Platform. Destroying a Terraform-managed CryptoKey will remove it from state and delete all CryptoKeyVersions, rendering the key unusable, but will not delete the resource from the project.

Google's current documentation disagrees — it says key rings, keys and key versions can be deleted in some circumstances, subject to preconditions: every version destroyed, no automatic rotation scheduled, not created by Autokey, and for a key ring, every key inside it gone first.

What actually happened. terraform destroy reported success and emptied its state — 0 secrets, 0 topics, 0 subscriptions, 0 functions, 0 service accounts, 0 buckets. The key ring and the key remained. Terraform did remove the automatic rotation schedule on its way out, which is one of Google's own preconditions for deleting a key, so part of the chain ran. Deleting the key immediately was refused with CRYPTOKEY_HAS_NON_DELETED_KEY_VERSIONS, because the version was DESTROY_SCHEDULED rather than destroyed.

So both sources are partly right. Google's preconditions are real and enforced exactly as written. The provider's “cannot be deleted” is too absolute — deletion is possible, just gated behind a wait this lab set to 24 hours and which defaults to 30 days.

Still unverified at publication: whether the key ring itself deletes once the version reaches DESTROYED. That cannot be known until the 24-hour window elapses, and this post is not going to predict it. The row stays Unverified until it has been run.

Key takeaways

1. “Rotation” in Secret Manager is a reminder, not an action. It publishes a message and stops. The component that changes the credential is one you write, and its absence looks exactly like an oversight rather than a design.
2. Under CMEK, the key holder is a kill switch — but not an instant one. Revocation took between 2 and 70 seconds across three trials, which is long enough that a naive test will call a working control broken.
3. A disabled key version is still a billed key version, and so is one scheduled for destruction. Only DESTROYED stops the meter, and the default wait for that is 30 days.
4. Enabling an API is not holding a permission, and attribution follows the credentials, not the resource. Both bit this week, three months apart from the weeks that set them up.
5. Your own guardrails are part of the platform now. Two controls from Weeks 03 and 09 blocked a Google service from doing something ordinary. Satisfying them was the right answer; weakening them would have been the easy one.

Repository

week-12-secret-management — Terraform, the rotator source, the measurement scripts and the lab guide.

References

• Secret rotation — the notification behaviour, the one-hour minimum, and the five-minute rule
• CMEK for Secret Manager — the global-key requirement for automatic replication
• Delete Cloud KMS resources — the deletion preconditions
• Destroy and restore key versions — the 30-day default
• Cloud KMS pricing — what counts as an active key version