Business Challenge
Every one of these follows from a single design choice: the Kubernetes object is not a description of the resource, it is the resource's controller.
That is the documented default. When you delete a Config Connector resource from the cluster, the associated Google Cloud resource is deleted by default. Not orphaned, not left behind — deleted. Which means kubectl delete namespace, an operation whose entire cultural history is that it destroys cluster state, now reaches production infrastructure.
Set cnrm.cloud.google.com/deletion-policy: abandon on anything stateful before it ever reaches a cluster, and treat namespace deletion as a change with a blast radius rather than a cleanup.
Working as designed. Config Connector reconciles continuously, so a console edit is drift and drift gets corrected. This is the behaviour Terraform users expect only at apply time, arriving on its own schedule instead.
Decide deliberately whether you want that. Where you do not, cnrm.cloud.google.com/reconcile-interval-in-seconds set to zero halts drift correction while still applying changes you make to the spec.
Some fields cannot be changed on a live resource, and Config Connector will not work around it: resources are not recreated when modifying immutable fields. Terraform would plan a destroy-and-recreate; this stops and tells you.
Correct approachAbandon the resource, change the field, and re-acquire it. That is the documented route, and it is why the abandon annotation is an everyday tool here rather than an emergency one.
It can, if you installed it that way. In cluster mode one identity serves everything; namespaced mode lets you divide permissions based on the respective concerns of different IAM service accounts, with a service account bound per namespace.
Correct approachChoose namespaced mode unless you have a reason not to. It is the difference between one credential for the whole estate and one per team.
Architecture
Config Connector is an open source Kubernetes add-on that lets you manage Google Cloud resources through Kubernetes. A Cloud SQL instance becomes a custom resource; a controller in the cluster makes the API calls; the object's status reports what happened.
The difference from the previous two posts
Terraform, self-managed or via Infrastructure Manager, is a batch process: it reads a description, compares it to a state file, and converges once, when invoked. Config Connector is a control loop. The desired state lives in the cluster's own database, and a controller reconciles it against the Google Cloud APIs asynchronously, on its own schedule, forever.
Almost everything surprising follows from that. There is no plan step to read before it acts. There is no state file, because etcd is the state. And there is no moment when reconciliation is finished, because it never is.
| Terraform (#25, #26) | Config Connector | |
|---|---|---|
| When it converges | When you run apply. | Continuously. |
| State | A file in a bucket. | The cluster. |
| Console change | Reported as drift at next plan. | Corrected without asking. |
| Delete | Explicit, from a plan you read. | Whatever deletes the object. |
| Identity | Your credentials, or a deploy service account. | A service account per namespace. |
Delete is the one to internalise
The associated Google Cloud resource is deleted by default. The annotation that changes it is one line:
metadata:
annotations:
cnrm.cloud.google.com/deletion-policy: abandonCompare the equivalent in post #26. Infrastructure Manager also lets you delete the deployment and keep the resources, but there the default is a considered choice at delete time and the abandoned resources become invisible to the tool. Here the default runs the other way: the resource goes unless you said otherwise, in advance, in a file.
Every safeguard a team has around kubectl delete was built when the worst outcome was a restarted pod. Namespace deletion is routine in most clusters — it is how you clean up a failed install, retire an environment, or reset a test. With Config Connector in namespaced mode, the namespace holds the objects that own your databases, buckets and networks. The habit and the consequence were formed in different worlds, and the annotation is the only thing between them.
Identity: the same question, a third time
Post #25 asked who can read the state bucket. Post #26 asked who can act as the deployment service account. Here the question is who can create objects in a namespace — because in namespaced mode each namespace is bound to its own IAM service account, and a dedicated controller pod impersonates it.
That is a genuinely good model. It maps cloud permissions onto Kubernetes RBAC, which teams already operate, and it makes the boundary a namespace rather than a credential file. The catch is the translation: a developer with create on a namespace has, transitively, whatever that namespace's service account can do in Google Cloud. RBAC that was scoped to "can deploy workloads" now also means "can provision infrastructure", and nothing in the RBAC binding says so.
Why This Architecture Holds Up
Drift correction is the feature and the hazard
A console change being reverted is exactly what you want for a security control and exactly what you do not want during an incident. At 3am somebody widens a firewall rule to restore service; the controller closes it again on its own schedule, and the person who made the change has no reason to suspect the cluster.
The lever is per-resource: cnrm.cloud.google.com/reconcile-interval-in-seconds, and setting it to zero halts drift correction while still applying changes made to the spec. Worth knowing before the incident rather than discovering it during one.
Immutable fields. Resources are not recreated when modifying immutable fields — the controller raises UpdateFailed and stops, where Terraform would plan a replacement. Safer, and it means abandon-and-reacquire is a routine procedure rather than a last resort. Unreadable fields. Mutable but unreadable fields are updated only when the custom resource is modified, because the controller cannot read them back to compare. So for that subset, drift is invisible and uncorrected — the guarantee is narrower than "the cluster is the truth" suggests.
Where this model earns its complexity
- You already run GKE and mean it. The cluster becomes a dependency of your infrastructure, not just a consumer of it. If the cluster is not treated as production, neither is your infrastructure.
- You want application and infrastructure in one manifest. A team shipping a service plus its bucket and database in one
kubectl applyis the case this was built for. - You want drift closed, not reported. Nothing in the Terraform posts does this. If unauthorised console changes are the problem you actually have, this is the tool that solves it.
Where it does not
- The estate is larger than the cluster's remit. Organization policies, folders, billing — the things posts #19 to #22 covered — sit oddly inside a workload cluster.
- You need a plan to review. There is no preview step here of the kind post #26 described. Approval has to happen in Git, before the object reaches the cluster, which makes the pull request the only gate.
- Your Kubernetes RBAC is not tight. The model inherits your cluster's access control. If that is loose, this makes it loose over your infrastructure too.
Key Architecture Decisions
| Decision | Choose this | Because |
|---|---|---|
| Stateful resources | deletion-policy: abandon, from the start |
The associated Google Cloud resource is deleted by default. |
| Namespace deletion | Treat as a production change | The namespace holds the objects that own the infrastructure. |
| Installation mode | Namespaced, unless you have a reason | It divides permissions across different IAM service accounts. |
| Cluster RBAC | Re-audit it as infrastructure access | Create in a namespace transitively means provision in Google Cloud. |
| Incident changes | Know the reconcile-interval annotation first | Setting it to zero halts drift correction while spec updates still apply. |
| Immutable field changes | Abandon, edit, re-acquire | Resources are not recreated when modifying immutable fields. |
| Trusting drift correction | Not for unreadable fields | Those update only when the custom resource is modified. |
| Change approval | In the pull request | There is no plan step between apply and action. |
| Organization-level resources | Keep them in Terraform | A workload cluster is the wrong owner for the estate's scaffolding. |
| Cluster criticality | Rate it as production infrastructure | Your infrastructure's control loop now lives in it. |
Closing Thought
Three posts, three tools, and the same question underneath each: what has the authority to change your infrastructure, and who has the authority to use it. A state bucket, a deployment service account, a Kubernetes namespace. The mechanism changes completely and the question does not move at all.
What is distinctive here is how far the authority travels from anything that looks like it. In post #25 the danger lived in a bucket, which at least looks like storage worth protecting. In #26 it lived in an IAM binding, which looks like nothing. Here it lives in a habit — the ordinary, unremarkable act of deleting a namespace, learned in an environment where the worst case was a rescheduled pod. Config Connector is a good model, and the drift correction is something neither Terraform post can offer. But it asks a team to unlearn something, and unlearning is the slowest kind of change there is.
#28 goes underneath all three: gcloud, the REST API, and what the console quietly does for you — the calls a click actually makes, and why a console-built resource is so hard to reproduce.
Comments