Auditing a resource hierarchy with Cloud Asset Inventory

Lab Overview

The problem. Six weeks of Terraform describe what this organization is supposed to contain. Nothing so far asks what it does contain — including the things created by hand, created before the lab existed, or created by someone who never read the plan.

Objective. Stand up an organization-wide asset feed for change notifications and a BigQuery snapshot for point-in-time queries, then run a real audit against the result.

Expected outcome. A queryable inventory of every asset in the organization, and at least one finding the Terraform could not have told us.

Cloud Services and Concepts

The concept that shapes the whole design: Google's services write as themselves. When Cloud Asset Inventory delivers to your topic or your dataset, the identity doing the writing is Google's own service agent for that product, not you and not your pipeline. Being project owner on the destination is irrelevant. You have to grant the service agent access to your own resources — which feels backwards until you have hit it once, and then you expect it everywhere.

Architecture and Workflow

The flow

1. Enable Cloud Asset Inventory and create a Pub/Sub topic and BigQuery dataset to receive output.
2. Grant Google's Cloud Asset service agent publisher rights on the topic and editor rights on the dataset — it writes as itself.
3. Create an organization-level feed, which publishes a message whenever any asset changes.
4. Run a snapshot export, which writes the full inventory to BigQuery.
5. Query the snapshot to audit the hierarchy — what sits outside a folder, what keys exist, what the Terraform never described.
6. Compare the findings against what the configuration claims.

Two decisions worth naming

The snapshot is a script, not a Terraform resource. An export is an operation with a moment attached; a feed is a desired state. There is no provider gap here — the two things are genuinely different, and forcing the export into Terraform would mean re-running it on every apply.

The feed is organization-scoped. A project-scoped feed would miss exactly the assets this audit exists to find: the ones nobody remembered to put in the right place.

Prerequisites

• An organization with several projects — ideally some created outside the hierarchy, so the audit has something real to find.
• roles/cloudasset.owner on the Terraform apply identity. This is a genuine over-grant and the lab says so — see Security.
• roles/cloudasset.viewer for the operator who runs the snapshot script.
• cloudasset, pubsub and bigquery APIs enabled on the quota project.
• Terraform 1.9+ with hashicorp/google ~> 6.0.

Week 1 supplies the hierarchy being audited. Exact grants, variables and queries are in the lab guide.

Implementation

Seven resources and one script. Commands and expected outputs are in the lab guide.

Step 1 — Create the destinations, then grant the service agent

Why: Cloud Asset Inventory writes as itself. Three of the seven resources exist solely to give Google's service agent access to your own topic, dataset and project.

service-<PROJECT_NUMBER>@gcp-sa-cloudasset.iam.gserviceaccount.com
  roles/pubsub.publisher      on the topic
  roles/bigquery.dataEditor   on the dataset
  roles/bigquery.jobUser      on the project

Validation: the topic's IAM policy names the service agent. Without it the feed is created successfully and then silently delivers nothing.

Step 2 — Create the organization feed

Why: organization-scoped, because a project-scoped feed would miss the assets that are not where they should be — which is the point of the audit.

Step 3 — Export a snapshot and audit it

Why: the feed answers "what changed"; the snapshot answers "what is there". The audit needs the second.

./scripts/snapshot.sh

Validation and Evidence

What the evidence proves. Rows one to four say the plumbing exists and the snapshot works. Row five is the actual result — the audit found two projects sitting directly under the organization, which no amount of reading the Terraform would have revealed. Row six is a near-miss worth keeping: the obvious query returns two keys and the correct answer is zero, because both are Google-managed. The last row is honest about an untested path: the feed's publisher grant was confirmed, but no change was made and no message observed.

Security and Cost

Security decisions

Cost

$0. Asset feeds and exports are free. The snapshot is a few kilobytes of BigQuery storage, well inside the free tier. Measured, not estimated.

Troubleshooting and Lessons

The same service gives an excellent error and a useless one

Problem: the BigQuery export failed with "The caller does not have permission" and nothing else.

Root cause: the Cloud Asset service agent lacked bigquery.dataEditor on the dataset. The identical class of mistake on the Pub/Sub side produces an error naming the service account, the permission and the resource.

Resolution: grant the service agent, not the caller.

Lesson: error quality varies within a single service. Do not assume a vague message means a subtle problem; it may be the same missing grant, reported badly.

No read-only role can read an asset feed

Problem: the plan phase could not refresh the feed resource.

Root cause: roles/cloudasset.viewer lacks cloudasset.feeds.get. Only cloudasset.owner has it, and that role also grants delete.

Resolution: the plan identity was given owner, and the hole was recorded rather than hidden. Week 8 replaces it with a two-permission custom role plus a deny policy.

Lesson: when no predefined role fits, say so in writing at the time. The alternative is an over-grant nobody remembers making.

The dashboard lied about freshness

Problem: the Asset Inventory console showed data six days old next to an export that had just completed.

Root cause: the dashboard is cached.

Lesson: query the snapshot table. A console page is a rendering, not the data.

Cleanup and Key Takeaways

Cleanup

terraform destroy removes all seven resources. Verify against the cloud: the feed should disappear from gcloud asset feeds list --organization=<ORG_ID>, and the topic and dataset should be gone.

Consider keeping it. Feeds and exports are free, and Week 8 is specifically about narrowing access to this feed — destroying Week 7 removes what Week 8 operates on. Commands are in the lab guide.

Ongoing charges: none.

Key takeaways

1. Google's services write to your resources as themselves. Grant the service agent, not the caller. Expect this for anything that delivers somewhere on your behalf.
2. An inventory answers questions a configuration cannot. Two projects sat outside the hierarchy; the Terraform had nothing to say about them because it never created them.
3. Filter on keyType. The obvious service-account-key query returns Google-managed keys too, so "two keys" became "zero user-managed keys" once filtered.
4. A feed is a state; an export is an operation. One belongs in Terraform and one belongs in a script, and conflating them means re-exporting on every apply.

References

• Lab guide — grants, snapshot, audit queries, known limitations
• Cloud Asset Inventory overview
• Monitoring asset changes
• Exporting to BigQuery
• Source repository