Should you run Argo CD, or let AWS?

Why — The Problem This Solves

Deploying to Kubernetes by hand does not scale past about one person. Somebody runs kubectl apply from a laptop, the cluster drifts away from what is in Git, and nobody can say with confidence what is actually running.

GitOps is the answer almost everyone reaches for. You stop applying things. You commit, and a controller inside the cluster notices, pulls the change, and makes reality match. Git becomes the record of what should be running, and the cluster converges on it. Argo CD is the tool most teams use to do that — a CNCF graduated project, running in a large share of production clusters.

So far, so standard. Here is what changed.

AWS now runs Argo CD for you. The EKS Capability for Argo CD puts the Argo CD controllers in AWS-managed infrastructure outside your cluster, authenticated through IAM Identity Center. Every tutorial you will find online installs Argo CD with Helm onto your own nodes, because until recently that was the only option.

That makes the interesting question no longer how do I install Argo CD. It is should I run it at all, or let AWS — and that is not a question a pricing page answers.

What this week actually does

Both, on one cluster, reading the same Git repository and deploying the same application. One variable: who operates Argo CD. Then a deliberate attempt to break the central GitOps promise — that the cluster converges on Git — to find where it stops being true.

What You Need to Know — Skills & Tools

Comfortable with Terraform, and the idea of a Kubernetes cluster. You do not need prior Argo CD experience; the vocabulary is small.

• GitOps — you do not deploy, you commit. A controller reconciles the cluster towards Git, continuously.
• Application — Argo CD's unit of work. It says: take this path in this repository and make this namespace on this cluster match it.
• Sync and self-heal — a sync applies Git to the cluster. Self-heal does it automatically when the two diverge, without waiting for a human.
• Prune — delete things Argo CD created that Git no longer declares. This one carries a trap, covered later.
• CRD (Custom Resource Definition) — how Kubernetes learns a new object type. Argo CD ships several. They are cluster-scoped, which matters more in this post than it sounds like it should.

Versions, checked the day this was built: Argo CD 3.5.3 (released 14 September 2026), Helm chart argo-cd 10.9.1, Kubernetes 1.36 on EKS, Terraform AWS provider 6.64.

Architecture — How It Fits Together

One cluster, one Git repository, and two controllers that disagree about where they should live.

Why both, which looks like indecision and is not

A comparison is only worth reading if the thing being compared is the only thing that differs. Same cluster, same repository, same application, same sync policy. What is left is the operating model — and the operating model decides things that have nothing to do with price.

How We Built It — Step by Step

Step 1 — Check the prerequisite that has no workaround

The managed capability authenticates through IAM Identity Center and nothing else. AWS is explicit that local Argo CD users are not supported. There is no admin password sitting in a Kubernetes secret to fetch, which is how essentially every Argo CD walkthrough begins.

No Identity Center instance, no managed Argo CD. Worth confirming before writing any Terraform:

Step 2 — Two modules, and one difference that is not cosmetic

The managed side is a single resource. The trust policy is the part worth reading closely, because the principal is not the one EKS clusters use:

resource "aws_eks_capability" "argocd" {
  cluster_name = var.cluster_name
  type         = "ARGOCD"
  role_arn     = aws_iam_role.capability.arn

  configuration {
    argo_cd {
      aws_idc { idc_instance_arn = var.idc_instance_arn }
      namespace = var.namespace
    }
  }
}

# capabilities.eks.amazonaws.com, not eks.amazonaws.com.
# The wrong principal fails with "Invalid IAM role".
data "aws_iam_policy_document" "capability_assume" {
  statement {
    actions = ["sts:AssumeRole", "sts:TagSession"]
    principals {
      type        = "Service"
      identifiers = ["capabilities.eks.amazonaws.com"]
    }
  }
}

That role gets no permissions policy at all. AWS's guidance is that Argo CD needs none by default — IAM permissions come in only for reading Git credentials from Secrets Manager or using CodeConnections. This week deploys from a public repository, so the role holds nothing beyond the ability to be assumed. An empty role that works tells you what the service actually needs; a broad role that works tells you nothing.

The self-managed side is the Helm chart, pinned so a rebuild six months from now is the same experiment:

resource "helm_release" "argocd" {
  chart     = "argo-cd"
  version   = "10.9.1"        # Argo CD 3.5.3
  namespace = "argocd-self"
}

And the two Application manifests differ in exactly one field:

# self-managed                    # managed capability
destination:                      destination:
  server: https://kubernetes.default.svc
                                    name: arn:aws:eks:us-east-1:...:cluster/week19-gitops

Upstream Argo CD runs inside the cluster, so "this cluster" is an address it can reach. The capability runs outside it, where "this cluster" is not a network location at all — it is an AWS resource, and AWS resources are named by ARN. The capability accepts only EKS cluster ARNs, and does not auto-register the local cluster. Every Argo CD example online needs that one edit before it will run here.

Step 3 — Deploy it

One workspace, VCS-driven, OIDC credentials, no static keys. The first apply failed, and the reason is the most useful thing in this post — it gets its own section below.

With both installed, the difference is not subtle:

Step 4 — Give the capability permission to do anything

Creating the capability auto-creates an EKS access entry for its role, and it is reasonable to assume that is enough. It is not, and the way it fails is worth knowing:

The symptom is a genuinely confusing pair of statuses: Sync Unknown, Health Healthy. Nothing is unhealthy because nothing was ever compared. Unknown here means could not look, not looks wrong.

Verifying it actually works

Both Applications reconciling, the workload running — and the two controllers disagreeing about who owns it:

Then the part the week was really for — four kinds of drift, to find where "the cluster converges on Git" stops being true:

In plain terms:

• A field Git specifies — reverted, in under three seconds.
• A field Git never mentions — survives. Self-heal has no target to converge it to.
• A resource created by hand — never pruned. It carries no tracking annotation, so it is not drift; it is furniture.
• A resource Git declares, deleted — recreated, as a genuinely new object.

Challenges — What Actually Went Wrong

1. The two Argo CDs cannot both own the CRDs

The first apply built the VPC, the cluster, the node group and the capability, then failed on the Helm release:

Unable to continue with install: CustomResourceDefinition
"applications.argoproj.io" in namespace "" exists and cannot be
imported into the current release: invalid ownership metadata;
label validation error: missing key "app.kubernetes.io/managed-by"

Note in namespace "". That is Kubernetes saying the object has no namespace, because it cannot have one. CRDs are cluster-scoped. Putting the two Argo CDs in separate namespaces isolates their Deployments, their Services and their configuration, and isolates nothing whatsoever about the type definitions they share.

AWS documents that creating the capability installs the CRDs. It is one sentence in a list of five, and I read straight past it.

Helm's refusal is correct, and worth understanding rather than working around. Adopting an object it did not create would mean a later helm uninstall deletes a CRD it does not own — and deleting a CRD deletes every custom resource of that kind, cluster-wide. In this cluster, uninstalling the Helm release would have silently taken the managed capability's Applications with it.

The fix is one line, and the meaning is larger than the line: crds.install = false. The second one in has to defer.

2. The documented RBAC fix binds to a group that does not exist

With the capability unable to read cluster state, AWS's documentation is clear about the remedy: bind a ClusterRole to the group eks-access-entry:PRINCIPAL_ARN. I did exactly that. Nothing changed.

$ aws eks describe-access-entry --principal-arn .../week19-gitops-argocd-capability
{
    "groups": [],
    "username": "arn:aws:sts::ACCOUNT:assumed-role/week19-gitops-argocd-capability/{{SessionName}}"
}

The auto-created access entry has no Kubernetes groups at all. The documented group name therefore matches nothing and the binding is inert. The group has to be added first:

aws eks update-access-entry --cluster-name week19-gitops \
  --principal-arn .../week19-gitops-argocd-capability \
  --kubernetes-groups argocd-capability

After that the Application moved from Unknown to OutOfSync — from "cannot look" to "looked, and here is the difference". AWS's quick alternative is AmazonEKSClusterAdminPolicy, which is system:masters; their own docs say not to use it in production, so this build used cluster-wide read plus namespace-scoped write instead.

3. My test was slower than the thing it was measuring

The drift script scaled the deployment from 2 replicas to 4, waited three seconds, and read the value back. It saw 2, and reported that the drift had never landed.

It had landed. Argo CD reverted it in under three seconds. The test was racing a controller whose entire job is to undo exactly that change, and losing.

The two obvious readings are both wrong and both comfortable: "the change didn't apply", or worse, "nothing happened, pass". The fix is to stop trying to catch a transient state and measure a fact that cannot be undone. metadata.generation increments on every write to spec and never decreases — it went 5 to 7, proving two writes, the scale and the revert. metadata.uid changes only when an object is genuinely destroyed and recreated, which is how the delete case is now proven rather than guessed at.

4. Two runs of the same test disagreed, and I nearly published the wrong explanation

One run said an out-of-band annotation survived. The next said it was removed. Same command, same cluster.

I had a tidy mechanism ready: Argo CD leaves unknown fields alone until something unrelated triggers a sync, and then the re-apply takes them with it. It is plausible, it makes a good paragraph, and a controlled experiment showed it was wrong — the annotation survived both when idle and across a sync that definitely fired.

The real cause was my own test. The previous run had ended by deleting the deployment; Argo CD's recreate landed while the next run was already measuring, so the annotation had been written to an object that no longer existed. A test that starts while the previous test's repair is still in flight is measuring the previous test. The script now refuses to begin until the object stops changing.

Security — Controls at Every Layer

• No static AWS credentials anywhere. HCP Terraform authenticates by OIDC; the capability role is assumed by capabilities.eks.amazonaws.com.
• The capability role holds no permissions policy. It needs none for a public repository, and an empty role that works is evidence about what the service actually requires.
• Least privilege over the documented shortcut. Cluster-wide read so Argo CD can discover and health-check; write scoped to the single namespace it deploys into — not system:masters.
• Identity Center, not local users. No shared admin password in a Kubernetes secret. Deploy authority is tied to identity you already govern, and revoking a person revokes their deploy access.
• Know who holds cluster admin before you need it. Whoever runs the automation becomes the cluster's creator-admin — and when that is a build runner, it is not you. Access entries are declared in Terraform for exactly that reason.

Cost

Prices as of September 2026 — verify at the EKS pricing page.

The comparison that changes the decision

The managed capability costs $0.03/hr. Upstream Argo CD is free software — and it needs somewhere to run. Seven pods do not fit beside kube-system on a t3.small's 2 GB, so the node goes to a t3.medium: +$0.0208/hr.

Those two numbers are close enough that cost is not the deciding factor — the opposite of what "managed AWS service versus free open source" usually implies. The decision gets made on the feature list instead, and the managed capability genuinely does less: no Config Management Plugins, no Notifications controller, no SSO provider other than Identity Center, no UI extensions, and a sync timeout fixed at 120 seconds that you cannot change.

Cleanup

./scripts/cleanup.sh

Eleven checks, all clear. Two are worth calling out, because they are the ones that quietly survive a careless teardown:

• The EKS Capability is an AWS resource, not a Kubernetes object. It bills hourly and is invisible to kubectl. A cluster that looks empty can still be charging.
• A deleted NAT gateway keeps its tags. A tag sweep finds it long after it stops billing, so the script resolves its state before reporting — a cleanup check that cries wolf is one people stop reading.

References

• Comparing EKS Capability for Argo CD to self-managed Argo CD — the unsupported-feature list, verbatim
• Amazon EKS capability IAM role — trust policy, and why no permissions are needed
• Register target clusters — ARNs not API server URLs, and the RBAC section
• Amazon EKS pricing
• Argo CD release support
• This week's Terraform, manifests and scripts

Key Takeaways

Namespaces do not separate CRDs. Two installations of the same operator in different namespaces still contend for one set of cluster-scoped type definitions. The second one in has to defer — and Helm is right to refuse rather than adopt.

"Unknown" is not "unhealthy". An Argo CD Application reporting Sync Unknown and Health Healthy has not found a problem; it has failed to look. Two very different situations wearing similar words.

Self-heal converges what it knows about. It reverts fields Git specifies and restores resources Git declares. It leaves alone any field Git never mentions, and it never prunes a resource it never tracked. "The cluster matches Git" does not mean "the cluster contains only what Git declares" — and that gap is where an out-of-band change hides indefinitely.

Two GitOps controllers pointed at the same resources will fight. The last writer takes the tracking annotation and reports Synced; the other reports OutOfSync forever, correctly.

Cost was not the deciding factor, and I expected it to be. $0.03/hr managed, against +$0.0208/hr of extra node to run free software. The real decision is the unsupported-feature list.

What I'd do differently in production

• Do not run both. This week ran two deliberately, to compare them. Pointing both at one namespace produced a permanent ownership conflict — pick one, or give them genuinely separate target clusters.
• Separate the hub from the workloads. The capability is designed for a hub cluster driving remote clusters through access entries, which sidesteps local-cluster registration entirely.
• Treat the 120-second sync timeout as a selection criterion. It is fixed and unchangeable. If a sync legitimately takes longer, the managed capability is not a candidate, and no amount of configuration will change that.
• Make drift tests prove their own preconditions. Anything else measures whatever the last test left behind.