Week 19 - GitOps on EKS: Should You Run Argo CD, or Let AWS?

Lab Overview

Deploying to Kubernetes by hand does not scale past about one person. Somebody runs kubectl apply from a laptop, the cluster drifts away from what is in Git, and nobody can say with confidence what is actually running.

The problem. GitOps is the usual answer: you stop applying things, you commit, and a controller inside the cluster pulls the change and makes reality match. Argo CD is the tool most teams use. But AWS now runs Argo CD for you — the EKS Capability puts the controllers in AWS-managed infrastructure outside your cluster. Every tutorial online installs it with Helm onto your own nodes, because until recently that was the only option.

The objective. Run both, on one cluster, against one repository, and find out what actually differs.

The outcome. The operating model decides things that have nothing to do with price — and the two installations cannot coexist as cleanly as separate namespaces suggest.

Difficulty	Advanced
Lab runtime	Ran 5h 19m before teardown (13:02–18:21 UTC)
Measured cost	$1.21 for 5h 19m
Deployment	Terraform via HCP Terraform with OIDC
Full lab guide	GitHub — prerequisites, Terraform, commands, validation, cleanup

Cloud Services and Concepts

Service	Purpose here	What it is
Amazon EKS	The cluster both controllers deploy into.	Managed Kubernetes. AWS runs the control plane and charges by the hour for it from the moment the cluster exists; the worker nodes and what runs on them are still yours.
EKS Capability for Argo CD	AWS-managed Argo CD, running outside your cluster with no node footprint, authenticated through IAM Identity Center.	Argo CD run by AWS outside your cluster, billed per hour and per application-hour. Its custom resource definitions are still installed into your cluster, where they are cluster-scoped and shared.
Argo CD (self-managed)	The same project installed by Helm onto your own nodes — the way every tutorial does it.	The upstream GitOps controller, installed into your own cluster by Helm. The software is free; the pods it runs are not.
IAM Identity Center	Who may deploy. Revoking a person revokes their deploy access.	AWS's sign-in and assignment layer for people. Users and groups are defined once, and assignments grant them a role in each account.
Terraform / HCP Terraform	Builds the cluster and both installations, with OIDC and no static keys.	Terraform declares infrastructure as code; HCP Terraform runs it on HashiCorp's machines and keeps the state file.

GitOps in one sentence: Git holds what should be running, a controller continuously makes the cluster match, and nobody applies anything by hand.

Two words that look alike and are not. An Argo CD Application reporting Sync Unknown and Health Healthy has not found a problem — it has failed to look. "Unknown" is not "unhealthy", and confusing them is how a broken sync reads as a working one.

Architecture and Workflow

One repository, two places to run the controllers, and the same CRDs in the cluster at the end of both.

One EKS cluster and one Git repository hold the desired state.
The AWS-managed capability runs its controllers outside the cluster, with no pods of yours, authenticated via Identity Center.
The self-managed Argo CD runs the identical project as pods on your own node group.
Both watch the same repository with the same sync policy and the same application.
Each reconciles the cluster toward Git, writing a tracking annotation on the resources it owns.
Where both manage the same resource, the last writer takes the annotation and reports Synced; the other reports OutOfSync permanently.

Design decisions

Both installations, deliberately. A comparison is only worth reading if the thing being compared is the only thing that differs: same cluster, same repository, same application, same sync policy. What is left is the operating model.

Least privilege over the documented shortcut. Cluster-wide read so Argo CD can discover and health-check; write scoped to the single namespace it deploys into — not system:masters.

Identity Center rather than local Argo CD users. No shared admin password in a Kubernetes secret.

Prerequisites

Requirement	Detail
AWS account	Permission for EKS, EC2, VPC, IAM and the EKS Capability
IAM Identity Center	An enabled instance — the capability authenticates through it
Terraform	>= 1.10, plus the Helm and Kubernetes providers
A Git repository	Public for this lab; the capability role needs no policy to read one
Region	`us-east-1`

Cost starts immediately. The EKS control plane bills $0.10/hr from creation with no free tier, and a NAT gateway adds $0.045/hr. Budget around $0.22/hr for the whole build and plan to destroy the same day.

Know who becomes cluster admin before you start. Whoever runs the automation becomes the cluster's creator-admin — and when that is a build runner, it is not you. Access entries are declared in Terraform for exactly that reason.

Implementation

A VPC, an EKS cluster and node group, the AWS-managed capability, and a Helm-installed Argo CD alongside it. The lab guide carries the full Terraform and the exact commands.

week-19-gitops-argocd/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the modules
|   +-- modules/cluster/    VPC, EKS, node group, access entries
|   +-- modules/capability/ the AWS-managed Argo CD and its IAM role
|   +-- modules/selfhosted/ the Helm release and its RBAC
|
+-- apps/                   the application both controllers deploy
+-- scripts/                drift tests and cleanup

Only the directories that matter. The full repository is on GitHub.

The capability role holds no permissions policy at all. It needs none for a public repository — and an empty role that works is evidence about what the service actually requires, rather than a guess padded for safety.

HCP Terraform run list for week-19-dev showing three runs: the initial plan errored, the retry applied, and the teardown applied, with Resources now 0 — Three runs, in order: the first apply **errored** on the CRD collision, the retry **applied**, the teardown **applied**. Resources 0. The teardown's title is a stale default from my own tooling that describes a different week entirely — noted here rather than quietly cropped out, and since fixed.

kubectl showing no pods in the argocd namespace, seven pods in argocd-self, the three argoproj CRDs, and both Applications — `kubectl get pods -n argocd` returns **No resources found**. The same command against `argocd-self` lists **seven**. Same product, same version family, and one of them is not in your cluster at all. What the capability *does* install is the three CRDs — which is exactly what broke the first apply.

Validation and Evidence

Deploying both controllers proves nothing about how they behave. The tests are about drift, ownership and what each one can actually see.

Test	Expected	Actual	Status
Both controllers deploy the app	Application running from Git	Both deployed it successfully	Pass
Self-heal reverts an edited field	Field restored to Git's value	Reverted	Pass
Self-heal restores a deleted resource	Recreated	Recreated	Pass
Self-heal removes an untracked resource	Pruned	Left alone indefinitely	Fail — by design, see below
Two controllers, one resource	One owner	Last writer takes the annotation; the other reports `OutOfSync` forever	Pass — correctly
Scoped RBAC is sufficient	Deploys without `system:masters`	Cluster-wide read plus one-namespace write was enough	Pass

The drift experiment output: eleven checks passed, none failed, none unmeasurable, across four kinds of drift — Eleven checks, nothing failed, and — after two rounds of fixing the test rather than the finding — nothing unmeasurable. The results are the same shape as "a namespace is not a boundary": the guarantee is real, and narrower than the slogan.

The tracking annotation showing the capability's namespace_appname format, and the two Applications reporting different resource statuses for the same objects — The deployment carries `argocd.argoproj.io/tracking-id: argocd_podinfo:...` — the capability's `namespace_appname` format. It reports all three resources Synced. The Helm install reports the same three OutOfSync. Neither is malfunctioning; both are correct about a cluster that has two owners.

Security and Cost

Security

No static AWS credentials anywhere. HCP Terraform authenticates by OIDC; the capability role is assumed by capabilities.eks.amazonaws.com.

The capability role holds no permissions policy, and works. For a public repository it needs none.

Least privilege over the documented shortcut. Cluster-wide read for discovery and health checks; write scoped to one namespace, not system:masters.

Identity Center, not local users. No shared admin password in a Kubernetes secret; revoking a person revokes their deploy access.

Know who holds cluster admin before you need it. Whoever runs the automation becomes creator-admin. On a CI runner, that is not you.

The capability's auto-attached access policies, the forbidden error on appprojects, the empty groups list on the access entry, and the Application moving from Unknown to OutOfSync after the group is added — The auto-created entry grants `AmazonEKSArgoCDClusterPolicy` at cluster scope and `AmazonEKSArgoCDPolicy` scoped to the `argocd` namespace *only* — enough for Argo CD to manage its own custom resources, not enough to deploy anywhere.

Cost

$1.21 measured from Cost Explorer on 19 September — not a calculation. At publication AWS had not posted the charges, so the section originally gave the rate and the duration and no total.

Line	Cost
EKS control plane	$0.51
NAT gateway ($0.27 hours, $0.03 data)	$0.30
t3.medium node	$0.21
Argo CD capability ($0.144 hours + $0.007 applications)	$0.15
Elastic IP, EBS, regional data transfer	$0.04

Ran 5h 19m at roughly $0.22/hr. The capability is the cheapest line on the bill — $0.03/hr plus $0.0015 per Application-hour, with no node footprint at all — which is precisely why price is the wrong axis to choose on. The control plane and the NAT gateway dominate either way.

Troubleshooting and Lessons

The two Argo CDs cannot both own the CRDs

Problem. The first apply built the VPC, cluster, node group and capability, then failed on the Helm release: "CustomResourceDefinition applications.argoproj.io in namespace "" exists and cannot be imported into the current release".
Root cause. Note namespace "" — that is Kubernetes saying the object has no namespace, because it cannot have one. CRDs are cluster-scoped. Separate namespaces isolate Deployments, Services and configuration, and isolate nothing whatsoever about the type definitions they share. Creating the capability installs the CRDs; AWS documents it in one sentence in a list of five.
Resolution. The second installation defers to the first rather than adopting the objects.
Lesson. Helm's refusal is correct and worth understanding rather than working around.

Self-heal converges what it knows about

Problem. An out-of-band resource survived indefinitely under a controller with self-heal enabled.
Root cause. Self-heal reverts fields Git specifies and restores resources Git declares. It leaves alone any field Git never mentions, and never prunes a resource it never tracked.
Resolution. Treat "tracked" as the unit, and prune explicitly if you need it.
Lesson. "The cluster matches Git" does not mean "the cluster contains only what Git declares" — and that gap is where an out-of-band change hides.

Two controllers pointed at the same resources will fight

Problem. One Application reported OutOfSync permanently.
Root cause. The last writer takes the tracking annotation and reports Synced; the other sees a resource it believes it owns, annotated for someone else.
Resolution. Give each controller its own resources. There is no shared-ownership mode.
Lesson. The permanently-unhappy controller is reporting correctly; the problem is the arrangement, not the report.

Cleanup and Key Takeaways

Eleven checks, all clear. Two are worth calling out, because they are the ones that quietly survive a careless teardown:

The EKS Capability is an AWS resource, not a Kubernetes object. It bills hourly and is invisible to kubectl. A cluster that looks empty can still be charging.
A deleted NAT gateway keeps its tags. A tag sweep finds it long after it stops billing, so the script resolves its state before reporting — a cleanup check that cries wolf is one people stop reading.

Key takeaways

Namespaces do not separate CRDs. Two installations of the same operator in different namespaces still contend for one set of cluster-scoped type definitions.
"Unknown" is not "unhealthy". An Application reporting Sync Unknown and Health Healthy has failed to look, not found nothing wrong.
Self-heal converges what it knows about — and never prunes a resource it never tracked.
Two GitOps controllers on the same resources will fight, and the loser reports OutOfSync forever, correctly.
Price is the wrong axis. The capability is the cheapest line on the bill; the control plane and NAT gateway dominate either way. Choose on operating model.

Should you run Argo CD, or let AWS?

📋 In This Post