S3 Intelligent Storage Platform: Tiering, Lifecycle, and Cost Automation

Why — The Problem This Solves

Every S3 bucket accumulates objects over time. Log files written daily, data exports from pipelines, backups, media uploads — they all go in at Standard storage rates and, without intervention, they all stay there at Standard rates indefinitely. Standard costs $0.023/GB/month. S3 Glacier Instant Retrieval costs $0.004/GB/month. That’s an 83% gap sitting there untouched on every cold object in your account.

The fix isn’t complicated: AWS already built the mechanism. S3 Intelligent-Tiering monitors individual object access patterns and moves objects between tiers automatically. S3 Lifecycle Policies transition objects on a schedule when the access pattern is already known. Neither requires any application changes. The cost is just that someone has to set it up — and in most environments, nobody does, because the objects keep working and the bill just quietly grows.

The Business Value

This platform delivers three things at once: automatic tiering that requires no application changes, deterministic lifecycle rules for data where the access pattern is already known, and a daily email report that shows exactly how much those two things saved versus doing nothing.

What You Need to Know — Skills & Tools

S3 Storage Classes

S3 has seven storage classes. The ones that matter here:

Prices verified 2026-06-30. Always check aws.amazon.com/s3/pricing/ for current rates.

Intelligent-Tiering vs. Lifecycle Policies

Key Concepts

Architecture — How It Fits Together

Two independent automation paths run on the same bucket. The event-driven path (S3 → SQS → Lambda) tags every new object. The scheduled path (EventBridge → Lambda → SNS) delivers the daily savings report.

Why SQS Between S3 and Lambda?

S3 supports direct Lambda triggers. The production reason to use SQS instead: at high object creation rates, direct S3 → Lambda triggers can throttle or drop events. SQS provides buffering — events queue up and Lambda processes them at its own pace. The DLQ gives you visibility when the tagger fails; without it, a tagging failure would silently disappear.

Why Lambda for Both Functions?

Both workloads fit Lambda’s constraints cleanly. The object tagger: event-triggered, stateless, completes in under a second, essentially zero base concurrency. The cost reporter: runs once per day, needs about 5 seconds to call CloudWatch and SNS, produces no output beyond the email. Neither workload has duration (ECS), parallelism (Batch), or orchestration (Step Functions) characteristics. Lambda is the right tool here — not because it was used in prior weeks, but because the workload shape matches.

How We Built It — Step by Step

Step 1 — Terraform Structure

Two Terraform modules under week-08-s3-intelligent-storage/terraform/modules/:

• s3-storage — the bucket, IT config, lifecycle rules, and bucket notification
• cost-automation — SQS, DLQ, both Lambdas, EventBridge rule, SNS topic, IAM roles, CloudWatch alarm

The two modules had a circular dependency: s3-storage needed the SQS queue ARN (from cost-automation), and cost-automation needed the bucket ARN (from s3-storage). Both ARNs are deterministic — computable from the account ID and naming convention without waiting for either resource to be created. Resolved in environments/dev/main.tf locals:

locals {
  account_id     = data.aws_caller_identity.current.account_id
  name_prefix    = "${var.project}-${var.environment}"
  bucket_name    = "${local.name_prefix}-${local.account_id}"
  bucket_arn     = "arn:aws:s3:::${local.bucket_name}"
  sqs_queue_name = "${local.name_prefix}-object-events"
  sqs_queue_arn  = "arn:aws:sqs:${var.aws_region}:${local.account_id}:${local.sqs_queue_name}"
}

Neither module output feeds the other — Terraform can plan and apply both in parallel with no dependency graph cycle.

Step 2 — S3 Intelligent-Tiering Configuration

IT is configured as a bucket-level rule (no prefix filter, so it covers all objects). Archive Instant tier activates at 90 days of no access; Deep Archive at 180 days. Objects under 128KB are automatically excluded by AWS and stay in Standard — the IT monitoring fee on small objects would exceed any storage savings.

resource "aws_s3_bucket_intelligent_tiering_configuration" "main" {
  bucket = aws_s3_bucket.main.id
  name   = "EntireBucket"

  tiering {
    access_tier = "ARCHIVE_ACCESS"
    days        = var.intelligent_tiering_archive_days  # 90
  }
  tiering {
    access_tier = "DEEP_ARCHIVE_ACCESS"
    days        = var.intelligent_tiering_deep_archive_days  # 180
  }
}

Step 3 — Lifecycle Policies

Three lifecycle rules on the same bucket:

• logs/ prefix rule — Standard-IA at 30d, Glacier Instant Retrieval at 90d, delete at 365d
• AbortIncompleteMultipartUpload — kills orphaned upload parts after 7 days (silent cost if skipped)
• NoncurrentVersionExpiration — expires old versions after 30 days (versioning is on; without this, every overwrite retains prior versions forever)

Step 4 — SQS + DLQ + Lambda: object_tagger

The SQS queue policy restricts SendMessage to the specific bucket ARN — no other principal can write to it. The Lambda reads from SQS, classifies the object by prefix (logs/, data/, media/, archive/) or file extension, and applies content-type and upload-date tags via PutObjectTagging. On failure, the message retries up to 3 times before landing in the DLQ.

def classify_object(key: str) -> str:
    key_lower = key.lower()
    if key_lower.startswith("logs/"):
        return "logs"
    if key_lower.startswith("data/"):
        return "data"
    if key_lower.startswith("media/"):
        return "media"
    if key_lower.startswith("archive/"):
        return "archive"
    if any(key_lower.endswith(ext) for ext in [".log", ".txt", ".csv"]):
        return "logs"
    return "data"

Step 5 — EventBridge + Lambda: storage_cost_reporter

An EventBridge rule fires once per day (rate(1 day)). The Lambda queries CloudWatch BucketSizeBytes with the StorageType dimension for each tier, computes actual cost, computes what the same bytes would cost at Standard, and publishes the difference to SNS as a plain-text email report.

Step 6 — Deploy via HCP Terraform

Backend is HCP Terraform Cloud (same pattern as weeks 5, 6, 7):

cloud {
  organization = "Katta"
  workspaces {
    name = "week-08-dev"
  }
}

Created workspace week-08-dev in HCP Terraform, connected to the GitHub repo via the existing GitHub App installation, and triggered the first apply from the HCP UI.

The plan showed 25 resources: the S3 bucket, IT config, lifecycle rules, SQS queue and DLQ, SQS queue policy, both Lambda functions, both IAM roles, EventBridge rule and target, SNS topic and subscription, CloudWatch alarm, and supporting permissions. First apply errored at 24/25 — see Challenges below.

After adding depends_on = [module.cost_automation] to the s3_storage module, re-applying created the remaining resource cleanly:

Step 7 — Verify S3 Configuration in the Console

After deploy, confirmed the bucket exists with the correct naming convention and verified both the Intelligent-Tiering configuration and lifecycle rules were applied:

Step 8 — Confirm SNS Subscription

The SNS email subscription requires a one-click confirmation from the target inbox. The daily reporter will not deliver until this is confirmed.

Step 9 — Test Object Tagging

Uploaded a test object to the logs/ prefix to trigger the SQS → Lambda pipeline. The object_tagger classified it by prefix and applied two tags:

echo "test log entry" | aws s3 cp - s3://$BUCKET/logs/app/2026-07-02.log --profile personal

Step 10 — Invoke the Cost Reporter Manually

Triggered the storage_cost_reporter Lambda manually to verify the function runs end-to-end. CloudWatch S3 metrics have a 24-hour delay, so the first invocation returned zero values — this is expected behavior, not a bug. The function returned StatusCode: 200 and a report noting no metric data yet:

aws lambda invoke \
  --function-name week-08-s3-storage-dev-storage-reporter \
  --invocation-type RequestResponse \
  out.json --profile personal && cat out.json

The next morning, the EventBridge daily schedule fired automatically and delivered the first real email report to the inbox — confirming the full end-to-end loop works:

Step 11 — Confirm EventBridge Schedule

Verified the EventBridge rule was created and enabled, confirming the daily schedule will fire automatically going forward:

Step 12 — Confirm object_tagger Invocations

Checked the object_tagger Lambda’s monitor tab to confirm the test upload triggered a successful invocation through the SQS queue:

Challenges — What Actually Went Wrong

Security — Controls at Every Layer

Cost

All prices us-east-1, verified 2026-06-30. Check aws.amazon.com/s3/pricing/ for current rates.

Cleanup

The repo includes week-08-s3-intelligent-storage/scripts/cleanup.sh which empties the bucket (all versions and delete markers) before running destroy. The week-08-destroy.yml GitHub Actions workflow calls this script automatically before terraform destroy.

To destroy, run the week-08-destroy.yml GitHub Actions workflow — it empties the bucket automatically before calling terraform destroy. Alternatively, trigger destroy directly from the HCP Terraform UI (week-08-dev workspace → Actions → Start destroy plan) after running the cleanup script first:

# Empty the versioned bucket before HCP destroy
BUCKET=$(aws s3api list-buckets --query "Buckets[?contains(Name,'week-08-s3-storage')].Name" --output text --profile personal)
bash week-08-s3-intelligent-storage/scripts/cleanup.sh $BUCKET

# Then trigger destroy from HCP Terraform UI: week-08-dev → Actions → Start destroy plan

References

• AWS Docs: S3 Intelligent-Tiering
• AWS Docs: S3 Lifecycle Management
• S3 Pricing (us-east-1, current)
• Terraform: aws_s3_bucket_intelligent_tiering_configuration
• Terraform: aws_s3_bucket_lifecycle_configuration
• AWS Docs: S3 Event Notifications
• Source Code: week-08-s3-intelligent-storage

Key Takeaways