EBS Savings Dashboard — Phase 1: Turning CUR 2.0 Billing Data Into a Live Cost Intelligence Platform

Why — The Problem This Solves

EBS volumes are one of the most reliably wasteful line items in any AWS bill — not because engineers are careless, but because the cost is invisible until someone actively looks for it.

In a typical enterprise AWS org with 500 accounts:

• Unattached volumes keep billing after the EC2 instance they served is terminated
• gp2 volumes never migrated to gp3 cost 20% more for identical throughput
• io1 volumes provisioned for peak workloads sit idle at $0.125/GB-month
• Snapshots pile up automatically and are never reviewed

I wanted a single dashboard that answers three questions at once:

1. How much are we spending on EBS across the entire org right now?
2. Which accounts and regions are the biggest offenders?
3. What is the projected saving if we clean up unused volumes and migrate gp2 → gp3?

Choosing the Right Approach

There are several ways to build cost visibility tooling on AWS. Here is how the main options compare:

The CUR 2.0 → Athena path wins because the data is already in your S3 bucket (AWS delivers it daily), Athena charges only for columns scanned ($5/TB), and the whole pipeline is serverless — nothing running when nobody is using it.

Architecture — How It Fits Together

Three swim lanes, entirely serverless, no persistent compute:

• ① Data Pipeline — CUR 2.0 Parquet lands in S3. Glue Catalog provides the schema. Athena queries the data. Results cached in a second S3 bucket for 3 days.
• ② API Layer — API Gateway triggers Lambda on every dashboard load. Lambda runs as an IAM Role with least-privilege permissions — no IAM users, no access keys anywhere.
• ③ Frontend — React app compiled and synced to a private S3 bucket. CloudFront serves it globally via OAC. Permanent HTTPS URL, free *.cloudfront.net certificate.

Step 1 — Enable CUR 2.0 Export

Everything starts here. CUR 2.0 is the billing export AWS delivers daily to an S3 bucket in Parquet format. It must be enabled from the management (payer) account — not a member account. Go to Billing → Data Exports → Create export.

The critical selections:

• Include Resource IDs — required. This is the EBS volume ID (vol-xxxxxxxx)
• Time granularity: Daily — not Hourly (24x more rows with zero benefit for monthly savings tracking)
• File versioning: Create new file — not Overwrite. You want historical months preserved so the trend chart has data to show
• Amazon Athena checked — AWS auto-creates the Glue Catalog table and Hive partitions. No Glue Crawler needed in production
• Parquet format — columnar, Snappy-compressed. Athena scans only the columns your SQL touches, not all 116

Step 2 — Data Pipeline: S3, Glue, Athena

Once CUR is configured, AWS delivers one Parquet file per billing period to your S3 bucket. For Phase 1 testing I generated 18 months of synthetic data using generate_mock_cur.py — simulating 500 accounts, 6 regions, and ~5,000 EBS volumes. This lets the full pipeline be validated before waiting 24 hours for real CUR delivery.

A Glue Crawler reads the bucket and registers the schema in the Glue Data Catalog. Athena then uses that catalog to query the files with SQL — without needing to know the physical file paths.

The Lambda filters to EBS-only rows using:

WHERE line_item_product_code = 'AmazonEC2'
  AND line_item_usage_type LIKE '%EBS:VolumeUsage%'

This skips EC2 instances, RDS, and every other service in the same CUR file — returning only EBS volume cost rows.

Athena Query Result Reuse

CUR data only updates once per day. Running a fresh Athena scan every time someone opens the dashboard is wasteful. Athena's Query Result Reuse returns cached results at $0 cost if the same query ran within the last 7 days. The first user each day pays ~$0.01 for the scan. Everyone else that day gets sub-second responses from cache.

Step 3 — Identity & Security (No IAM Users)

One design principle held throughout: no IAM users anywhere. Every action is performed by IAM Roles with temporary STS credentials that auto-expire in 15 minutes.

The IAM role policy is scoped to exactly what Phase 1 needs:

Athena:   StartQueryExecution, GetQueryExecution, GetQueryResults
S3 CUR:   GetObject, ListBucket  (read-only)
S3 Res:   PutObject, GetObject, DeleteObject
Glue:     GetTable, GetDatabase, GetPartitions
# No EC2, no STS AssumeRole, no Organizations — CUR data only

Step 4 — Lambda & API Gateway

Lambda (handler_phase1.py) is the engine of the dashboard. It receives the API Gateway request, builds a parameterised Athena SQL query, polls for results, and returns a structured JSON payload to the browser.

Key design decisions in the Lambda:

• Region filter — ?region=us-east-1 appended to the fetch URL threads through to a {region_filter} placeholder in the SQL WHERE clause
• Months selector — 3/6/12/18 months controls how far back the Athena query looks
• Split-month logic — divides data into pre-cleanup and post-cleanup periods to calculate savings delta
• Division-by-zero guard — returns zero savings (not an error) when there is no pre-cleanup data

API Gateway is configured as an HTTP API (not REST) with CORS open for Phase 1 testing. Before connecting real CUR data at work, add IAM authorization or an API key so the endpoint is not publicly callable.

Step 5 — React Dashboard on CloudFront

The React app (AWS CloudScape Design System) is compiled with npm run build and synced to a private S3 bucket. CloudFront sits in front of it via OAC (Origin Access Control) — the S3 bucket has no public access at all. Every request is served over HTTPS from the free *.cloudfront.net certificate.

Deploying a new version is three commands:

npm run build
aws s3 sync dist/ s3://ebsp1-frontend-684346483786/ --delete --profile personal
aws cloudfront create-invalidation \
  --distribution-id E3W08TQPVK17E8 \
  --paths "/*" --profile personal

The CloudFront URL is permanent — it does not change when you redeploy. Share it once with your FinOps team and it always points to the latest version.

What the Dashboard Shows

The dashboard has four tabs, all responding to a global region filter and months selector (3 / 6 / 12 / 18 months).

Challenges & Real Bugs Fixed

Cost to Run

At enterprise scale (real CUR, 500 accounts, 50 engineers opening the dashboard daily), Athena cost rises to ~$2–5/month due to larger Parquet files. Everything else stays the same.

Switching to Real CUR Data (3 Steps)

Phase 1 runs on synthetic data. When your platform team enables real CUR, only three things change. Lambda, API Gateway, CloudFront, and the React dashboard are untouched.

What Phase 2 Adds

Phase 1 answers "how much are we spending." Phase 2 will answer "which specific volumes should we delete today."

Phase 2 adds cross-account EC2 inventory via STS AssumeRole. Lambda will temporarily assume an EBSDashboardReadRole in each member account and call EC2:DescribeVolumes to pull live attachment status, volume state, last-attached time, and tags.

A CloudFormation StackSet (deployed via Terraform) automatically creates EBSDashboardReadRole in every member account — including new accounts added to the org in future. The trust policy uses a PrincipalOrgID condition so only Lambda inside the same AWS Organization can assume the role. No IAM users, no long-lived credentials — same security model as Phase 1.

This unlocks a fifth tab: Volume Inventory — a sortable table of every unattached volume with size, type, region, account, last-attached date, and projected monthly saving if deleted.

Key Takeaways

What’s Next

• Add API Gateway authorization (IAM auth or API key) — current endpoint is open for testing only
• Add CloudWatch alarms for Lambda error rate and p99 duration
• Add EventBridge daily pre-warm schedule (06:00 UTC)
• Connect real CUR 2.0 export from management account
• Phase 2: StackSet deployment + live Volume Inventory tab

Full Terraform, Lambda handler, React dashboard, and architecture draw.io diagrams are in the GitHub repo.