📋 In This Post
Lab Overview
Someone turns on flow logs account-wide during a compliance push. The logs land in a bucket. Nobody has permission to read that bucket, and nobody has built a table over it, so nothing is ever queried.
The problem. Months later a NAT bill quadruples, or a host starts talking to somewhere it should not. The answer is technically sitting in S3. Getting it out means writing a Glue table definition, working out a partition strategy, and hand-writing SQL against a 30-field record format — under incident pressure, by someone who has not done it before. Flow logs are pure cost until someone builds the query layer, and the query layer is the part that always gets deferred. You pay $0.25 per GB ingested whether or not a single query ever runs.
The objective. Build the query layer: a small VPC that produces identifiable traffic, capture into Parquet on S3, and a query layer with a hard spend ceiling.
The outcome. It works — and two of the three things that went wrong produced zero rows and a green "succeeded", which is indistinguishable from a working system.
| Difficulty | Advanced |
| Lab runtime | About 48 hours before teardown |
| Cost | Estimated ~$3 for the build; ~$33/month if left running |
| Deployment | Terraform via HCP Terraform with OIDC |
| Full lab guide | GitHub — prerequisites, Terraform, commands, validation, cleanup |
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| VPC Flow Logs | The raw record of who talked to whom. Delivered to S3 in Parquet, not to CloudWatch Logs — half the ingest cost and the only path to columnar storage. | A record of every network flow a VPC accepted or rejected — addresses, ports, bytes and the verdict. Record version 11 adds fields such as the instance tag and the next-hop family. |
| Amazon S3 | Where the logs land, with Hive-style partitioning so Athena can skip what it does not need. | Object storage: you put files (objects) into buckets and pay for what you store and for each request, with no disk to size in advance. |
| AWS Glue Data Catalog | The table definition over those logs, using partition projection rather than a crawler. | A catalogue of table definitions — names, columns and types — that query engines read so they know how to interpret files in S3. |
| Amazon Athena | The query layer, inside a workgroup carrying a hard per-query scan ceiling. | Runs SQL straight against files in S3 with no database to run. You are billed for the bytes each query scans, so file layout is a cost decision. |
| NAT Gateway | Both a cost centre and a subject: attributing its charges is one of the week's findings. | Lets a private subnet reach the internet outbound while staying unreachable inbound. It is billed per hour AND per gigabyte, which is what makes it the expensive component in most small builds. |
| CloudWatch | Anomaly detection where normal is unknown, static thresholds where zero is the right answer. | AWS's metrics and logs service. Most services publish numbers into it automatically, without being asked to. |
Two fields that look interchangeable and are not.
traffic_path describes the path a packet took within AWS;
next_hop_interface_type names what the traffic actually went through. Only the
second can attribute NAT gateway cost, and the first is the one that sounds right.
Partition projection computes partitions from the S3 path rather than recording them. For flow logs that matters: the alternative is a crawler you pay to rediscover a layout that is fixed and documented.
Architecture and Workflow
- A generator instance in a private subnet reaches the internet through the NAT gateway, and reaches S3 through a gateway endpoint. Same instance, two destinations, two very different prices.
- An exposed instance in the public subnet has a security group with no ingress rules at all. It is internet-reachable and nothing can connect to it.
- Those refused connections produce the REJECT records the detection queries analyse. Removing them would not harden anything — it would delete the signal.
- Flow logs capture to S3 in Parquet, Hive-partitioned by date.
- A Glue table with partition projection makes it queryable without a crawler.
- Athena queries it inside a workgroup with a per-query scan
ceiling, so a careless
SELECT *cannot run up a bill. - A daily analyzer publishes metrics; CloudWatch alarms watch them.
Design decisions
S3, not CloudWatch Logs. Half the ingest cost, and the only route to Parquet. CloudWatch Logs is right only if you specifically need metric filters on individual flows.
A spend ceiling on the workgroup, not a convention. Athena has no default limit. The ceiling is enforced before anyone else touches the account.
Mixed alarm types, on purpose. Anomaly detection where normal is genuinely unknown; static thresholds where zero is the right answer. Using anomaly detection everywhere costs about 30× more and detects less.
Prerequisites
| Requirement | Detail |
|---|---|
| AWS account | Permission for VPC, EC2, S3, Glue, Athena, Lambda, CloudWatch |
| Terraform | >= 1.10 |
| HCP Terraform | Workspace with OIDC to AWS. No static keys |
| Instance access | SSM Session Manager — no SSH, no key pairs |
| Region | us-east-1 |
Check which availability zone you land in.
data.aws_availability_zones with state = "available" also returns
opted-in Local Zones, which carry a deliberately reduced subset of regional
services. One wrong AZ can look like three unrelated failures — see Troubleshooting.
Filter on zone type.
The NAT gateway is the cost. About $33/month if left running, roughly three quarters of the total, and it bills whether or not a single packet flows. On this build, forgetting is the risk — not the rate.
Implementation
A small VPC built specifically to produce identifiable traffic, flow logs to S3 in Parquet, a Glue table, an Athena workgroup with a ceiling, and a daily analyzer with alarms. The lab guide carries the full Terraform and every query.
week-14-vpc-flow-logs-intelligence/
|
+-- terraform/
| +-- environments/dev/ HCP backend, OIDC, wires the modules
| +-- modules/network/ VPC, subnets, NAT, the two traffic generators
| +-- modules/capture/ flow logs to S3, Parquet, Hive partitioning
| +-- modules/catalog/ Glue table with partition projection
| +-- modules/analyzer/ workgroup ceiling, queries, Lambda, alarms
|
+-- queries/ the detection and attribution SQL
+-- scripts/ teardown verification
One configuration field halves the cost — delivering to S3 rather than CloudWatch Logs. It is set once and never mentioned again, which is exactly why it is worth saying out loud.
Validation and Evidence
Every failure mode in this build is silent: zero rows and a green "succeeded" look identical to a working system. So each check asks for a number, not a status.
| Test | Expected | Actual | Status |
|---|---|---|---|
| Partitions resolve | Rows returned for today | Returned | Pass |
| Both traffic paths visible | NAT and gateway-endpoint traffic distinguishable | Distinguishable — via next_hop_interface_type | Pass |
NAT cost attribution via traffic_path | Rows naming NAT traffic | Zero rows; analyzer published nat_bytes = 0 | Fail — wrong field |
| REJECT records captured | Refused connections appear | Appeared, at scale | Pass |
| Port-scan alarm fires | Notification sent | Fired and delivered | Pass |
| Scan ceiling enforced | An oversized query is refused | Refused at workgroup level | Pass |
Security and Cost
Security
One deliberate exception runs through this build: the exposed instance is internet-reachable on purpose. Its security group has no ingress rules at all, so nothing can connect — and those denials are what generate the REJECT records the detection queries analyse. It holds no data and was destroyed with the rest of the stack.
No SSH, no key pairs, no inbound ports. Instance access is SSM Session Manager only. IMDSv2 required on both instances, encrypted root volumes.
Bucket hardening: all public access blocked,
BucketOwnerEnforced, SSE-S3 with bucket keys, and a policy denying any non-TLS
request.
Confused-deputy protection: the log delivery statement is scoped by both
aws:SourceAccount and aws:SourceArn, so another account cannot point
its flow logs at this bucket.
Least privilege. The analyzer's Athena permissions are scoped to the one workgroup carrying the scan ceiling — a broad grant would let it bypass the guardrail the design depends on.
Cost
These are rates and an estimate, not a measurement. This post was published before the charges posted and no Cost Explorer figure was recorded afterwards, so it says which it is rather than letting arithmetic read as a bill.
| Item | Rate | 48-hour build |
|---|---|---|
| NAT Gateway | $0.045/hr + $0.045/GB | ~$2.16 |
| 2 × t4g.nano | $0.0042/hr each | ~$0.40 |
| Flow log ingest to S3 | $0.25/GB | < $0.10 |
| Athena | $5/TB scanned | cents |
| Anomaly alarms × 2 | $0.30/month each | ~$0.02 |
Left running: about $33 a month, roughly three quarters of it NAT gateway. The NAT bills whether or not a single packet flows and produces no usage signal to remind you it exists.
The cost controls are in the code rather than in advice: a 30-day expiry on raw logs, a noncurrent-version expiry alongside it (versioning is on, so expiry alone would create delete markers while the billed bytes stayed), and a multipart-upload abort for orphaned partial writes.
Troubleshooting and Lessons
One wrong availability zone, wearing three disguises
Problem. The first apply failed with three unrelated-looking errors at
once: NatGateway NotAvailableInZone, gp3
VolumeTypeNotAvailableInZone, and t4g.nano not offered.
Root cause. One cause.
data.aws_availability_zones with state = "available" also returns
opted-in Local Zones. This account has the Dallas Local Zone enabled, so
names[0] resolved to us-east-1-dfw-1a, and Local Zones carry a
deliberately reduced subset of regional services.
Resolution. Filter on zone type and intersect with the instance type's actual
offering list. Worth keeping the second half even with the filter: among genuine AZs,
us-east-1e offers neither Graviton nor gp3.
Lesson. Three errors in three services, one cause. Check which AZ you landed
in before debugging any of them.
traffic_path cannot attribute NAT cost
Problem. The NAT attribution query returned zero rows and the analyzer
published a permanent nat_bytes = 0. Both looked like a broken pipeline.
Root cause. traffic_path describes the path a packet took within
AWS. It is not the field that names what traffic went through.
Resolution. next_hop_interface_type.
Lesson. The field that looks right is measuring something
else — and a wrong field returns zero rows, not an error.
Every failure here is silent
Problem. A broken table, a wrong field and an empty partition all present
identically: zero rows, query succeeded.
Root cause. Athena reports on the query, not on whether the answer is
plausible.
Resolution. Verify against delivered data rather than against your
configuration — count rows for a window you know had traffic.
Lesson. No alarm fires for "your logs are unqueryable."
Cleanup and Key Takeaways
The workspace is VCS-connected, which blocks terraform destroy from a CLI
checkout but not a destroy queued through HCP. The verification script does not destroy
anything; it checks what survived, and distinguishes three outcomes rather than
two — gone, still present, and could not verify. An expired session
returning an empty result otherwise reads as a clean teardown while having checked nothing.
What tends to linger, and why:
- The NAT gateway — the expensive one, with no usage signal to remind you it exists.
- The Elastic IP — billed hourly once detached, and it outlives the NAT gateway it belonged to.
- Anomaly detectors — a separate resource that survives a careless destroy.
Key takeaways
- Pick S3, not CloudWatch Logs, unless you specifically need metric filters on individual flows. Half the ingest cost, and the only path to Parquet.
- Verify against delivered data, not against your config. Every failure mode here is silent — zero rows and a green "succeeded" look identical to a working system.
- Do not filter on
traffic_pathto find NAT cost. Usenext_hop_interface_type. - Put a scan ceiling on the workgroup before anyone else touches it. Athena has no default spend limit.
- Check which AZ you landed in.
data.aws_availability_zonesreturns Local Zones too, and one wrong AZ can look like three unrelated failures.
Comments