Week 5 - Cost Anomaly Detection with AWS Cost Explorer, SNS, and Lambda

1. The Challenge — Why Fixed Budget Alerts Aren’t Enough

You set a monthly AWS budget of $500. You get an alert when you hit $450. Job done, right?

Not quite. Here’s the scenario that breaks that model: your EC2 spend is normally $80/month. This week it’s running at $200 — 150% above normal. But your total bill is still $280, well under your $500 alert threshold. No alert fires. No one notices. The overrun continues for days.

Fixed budget alerts have four fundamental blind spots:

• Threshold blindness. A fixed dollar limit can’t distinguish between $200 spend in a month where $200 is normal versus $200 when $50 is normal.
• Lagging detection. Budget alerts fire after a percentage of the monthly budget is consumed — which may be days after the anomalous spending began.
• No root cause. A budget alert tells you how much you’ve spent. Cost Anomaly Detection tells you which service, which region, which usage type is the culprit.
• No baseline learning. Budget thresholds don’t account for your actual spending patterns. The ML model updates itself as your workloads grow.

2. Why Implement Cost Anomaly Detection

AWS Cost Anomaly Detection uses ML to model your expected spend per service per day. It triggers an alert the moment actual spend diverges significantly from that model — regardless of whether a fixed budget threshold was crossed.

3. Skills & Tools Required

AWS Services

• AWS Cost Explorer & Cost Anomaly Detection
• SNS (Simple Notification Service) — two-topic fan-out pattern
• Lambda (Python 3.12)
• IAM — least-privilege roles, OIDC federation
• CloudWatch Logs

Infrastructure & Tooling

• Terraform — modules, cloud {} backend block
• HCP Terraform — VCS-driven workspaces, OIDC dynamic credentials
• Python — boto3, json, os
• GitHub — VCS trigger for HCP runs

Concepts to understand before starting

• ML-based anomaly detection. The model needs 10+ days of data to build a reliable baseline. Don’t expect alerts on a fresh account immediately.
• SNS fan-out pattern. Two topics: one receives raw JSON from AWS, the second delivers a formatted email. Lambda sits between them as a transformer.
• OIDC federation. HCP Terraform authenticates to AWS using short-lived tokens — no static keys stored anywhere.
• VCS-driven IaC. Push to GitHub → HCP plans automatically. Merge to main → HCP applies. No GitHub Actions Terraform steps needed.

4. Architecture

The architecture uses two SNS topics by design. The raw-anomaly topic receives dense JSON from AWS — not human-readable. Lambda transforms it and publishes a formatted plain-text alert to the cost-alert topic, which delivers to email subscribers. This separation gives a clean extension point: add Slack, PagerDuty, or Teams as additional subscribers to the formatted topic later without touching the detection layer.

5. How to Implement — Step by Step

Step 1 — Create the AWS OIDC provider and IAM role

Before HCP can authenticate to AWS, AWS needs to trust HCP. This is a one-time, manual bootstrap — it can’t be created by the same Terraform run that depends on it.

Register app.terraform.io as an OIDC identity provider, then create the role HCP will assume:

aws iam create-open-id-connect-provider \
  --url https://app.terraform.io \
  --client-id-list aws.workload.identity \
  --thumbprint-list <thumbprint>

aws iam create-role \
  --role-name hcp-terraform-role \
  --assume-role-policy-document file://trust-policy.json

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/app.terraform.io"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "app.terraform.io:aud": "aws.workload.identity"
      },
      "StringLike": {
        "app.terraform.io:sub":
          "organization:katta:project:*:workspace:week-05-dev:run_phase:*"
      }
    }
  }]
}

Step 2 — Set up HCP Terraform workspace

Previous weeks used an S3 backend. This week switches to HCP Terraform — a managed backend with VCS integration and a cleaner OIDC credential story.

Create the workspace in app.terraform.io: org Katta → New Workspace → Version Control Workflow → connect katta698/AWS-Platform-Engineering-Lab → working directory week-05-cost-anomaly-detection/terraform/environments/dev → name it week-05-dev.

Step 3 — Terraform backend and module composition

terraform {
  cloud {
    organization = "Katta"
    workspaces { name = "week-05-dev" }
  }
}

module "sns"           { source = "../../modules/sns"           ... }
module "lambda_alerter"{ source = "../../modules/lambda_alerter" ... }
module "cost_anomaly"  { source = "../../modules/cost_anomaly"   ... }

Step 4 — SNS topic policy (the part everyone gets wrong)

Without this policy, costalerts.amazonaws.com cannot publish to the topic and no alerts will ever arrive. There is no error message — AWS silently drops the publish attempt.

resource "aws_sns_topic_policy" "raw_anomaly" {
  arn = aws_sns_topic.raw_anomaly.arn

  policy = jsonencode({
    Statement = [{
      Principal = { Service = "costalerts.amazonaws.com" }
      Action    = "SNS:Publish"
      Resource  = aws_sns_topic.raw_anomaly.arn
    }]
  })
}

Step 5 — Cost Anomaly Monitor and Subscription

resource "aws_ce_anomaly_monitor" "this" {
  name              = "${var.project}-${var.environment}-monitor"
  monitor_type      = "DIMENSIONAL"
  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "this" {
  frequency        = "IMMEDIATE"
  monitor_arn_list = [aws_ce_anomaly_monitor.this.arn]

  subscriber { type = "SNS"; address = var.raw_sns_topic_arn }

  threshold_expression {
    dimension {
      key           = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
      values        = ["10"]
      match_options = ["GREATER_THAN_OR_EQUAL"]
    }
  }
}

Step 6 — Lambda alert formatter

Lambda transforms the raw Cost Anomaly Detection JSON payload into a plain-text email showing the service, expected vs. actual spend, and the top 5 root causes.

def format_alert(payload):
    impact   = payload.get("impact", {})
    expected = impact.get("totalExpectedSpend", 0.0)
    actual   = impact.get("totalActualSpend",   0.0)
    overage  = impact.get("totalImpact",        0.0)
    pct      = impact.get("totalImpactPercentage", 0.0)
    service  = payload.get("dimensionValue", "Unknown")

    root_causes = "\n".join(
        f"  - {rc['service']} | {rc['region']} | {rc['usageType']}"
        for rc in payload.get("rootCauses", [])[:5]
    )
    subject = f"[DEV] Cost Anomaly: ${overage:.2f} above expected - {service}"
    return subject, body

Step 7 — Configure OIDC dynamic credentials

Two workspace environment variables replace the entire AWS credential setup, pointing at the role created in Step 1. HCP mints short-lived tokens per run via sts:AssumeRoleWithWebIdentity — no static keys stored anywhere.

TFC_AWS_PROVIDER_AUTH  = true
TFC_AWS_RUN_ROLE_ARN   = arn:aws:iam::<ACCOUNT_ID>:role/hcp-terraform-role

Step 8 — Push to GitHub and apply via HCP

A git push to the connected branch automatically queues a plan in HCP — no manual trigger needed for that part. But a VCS-connected workspace blocks terraform apply from local CLI or CI entirely (Error: Apply not allowed for workspaces with a VCS connection). The actual resource creation only happens once you open the run in the HCP UI and click Confirm & Apply.

Step 9 — Test immediately after deploy

Don’t wait for a real anomaly. Use the Lambda test console with a mock payload to verify the full pipeline:

{
  "Records": [{
    "Sns": {
      "Message": "{\"dimensionValue\":\"Amazon EC2\",
                   \"impact\":{\"totalActualSpend\":22.50,
                               \"totalExpectedSpend\":10.00,
                               \"totalImpact\":12.50},
                   \"rootCauses\":[{\"service\":\"Amazon EC2\",
                                    \"region\":\"us-east-1\",
                                    \"usageType\":\"BoxUsage:t3.medium\"}]}"
    }
  }]
}

6. Real Challenges & Gotchas

7. Security Best Practices

8. Cost & Cleanup

Workspace (week-05-dev) → Actions → Start new run → Plan and destroy
→ review the plan → Confirm & Apply

9. References

• AWS DocsGetting started with AWS Cost Anomaly Detection — monitor types, subscription frequencies, threshold expressions.
• AWS DocsReceiving Cost Anomaly Detection alerts using Amazon SNS — documents the costalerts.amazonaws.com service principal.
• Terraformaws_ce_anomaly_monitor — DIMENSIONAL and CUSTOM monitor types.
• Terraformaws_ce_anomaly_subscription — threshold expression syntax and subscriber configuration.
• HCP TerraformDynamic Provider Credentials — AWS — OIDC trust setup between HCP and AWS.
• GitHubFull source code — week-05-cost-anomaly-detection

10. Key Takeaways

• costalerts.amazonaws.com is the SNS publisher principal for Cost Anomaly Detection — wrong principal causes silent failure with no error.
• The ML baseline needs 24–48 hours on a new monitor. Test with Lambda test events immediately after deploy.
• Two SNS topics is the right pattern — raw topic for AWS-to-Lambda, formatted topic for Lambda-to-human. Gives a clean extension point for Slack or PagerDuty later.
• HCP dynamic credentials eliminate static AWS keys entirely. 15 minutes of setup, zero ongoing rotation burden.
• Confirm the SNS email subscription before testing. Messages to unconfirmed subscriptions are silently dropped.
• VCS-connected HCP workspaces fail in confusing, layered ways — GitHub App repo access, OIDC sub case-sensitivity, and stale config ingestion can all produce the exact same generic error. Check "Include submodules on clone" early if a VCS-connected workspace can't find files that genuinely exist in the repo.
• Cost Anomaly Detection is completely free. The only reason not to have it in every AWS account is not knowing it exists.