Home› Blog› Week 14 - VPC Flow Logs + Network Intelligence: Th…
AWS Weekly Lab AWS Terraform

The logs everyone enables and nobody reads

VPC Flow Logs get switched on for a compliance checkbox, write tens of gigabytes a month into a bucket nobody can query, and are never opened again — until an incident, when the answer is technically in S3 and unreachable in under a day. This week builds the layer that makes them answer questions, and finds out the hard way that the field everyone reaches for cannot do the job.

Verified against current vendor documentation on 22 August 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.
AWS Platform Engineering Lab · Week 14 of 52

Lab Overview

Someone turns on flow logs account-wide during a compliance push. The logs land in a bucket. Nobody has permission to read that bucket, and nobody has built a table over it, so nothing is ever queried.

The problem. Months later a NAT bill quadruples, or a host starts talking to somewhere it should not. The answer is technically sitting in S3. Getting it out means writing a Glue table definition, working out a partition strategy, and hand-writing SQL against a 30-field record format — under incident pressure, by someone who has not done it before. Flow logs are pure cost until someone builds the query layer, and the query layer is the part that always gets deferred. You pay $0.25 per GB ingested whether or not a single query ever runs.

The objective. Build the query layer: a small VPC that produces identifiable traffic, capture into Parquet on S3, and a query layer with a hard spend ceiling.

The outcome. It works — and two of the three things that went wrong produced zero rows and a green "succeeded", which is indistinguishable from a working system.

DifficultyAdvanced
Lab runtimeAbout 48 hours before teardown
CostEstimated ~$3 for the build; ~$33/month if left running
DeploymentTerraform via HCP Terraform with OIDC
Full lab guideGitHub — prerequisites, Terraform, commands, validation, cleanup

Cloud Services and Concepts

ServicePurpose hereWhat it is
VPC Flow LogsThe raw record of who talked to whom. Delivered to S3 in Parquet, not to CloudWatch Logs — half the ingest cost and the only path to columnar storage.A record of every network flow a VPC accepted or rejected — addresses, ports, bytes and the verdict. Record version 11 adds fields such as the instance tag and the next-hop family.
Amazon S3Where the logs land, with Hive-style partitioning so Athena can skip what it does not need.Object storage: you put files (objects) into buckets and pay for what you store and for each request, with no disk to size in advance.
AWS Glue Data CatalogThe table definition over those logs, using partition projection rather than a crawler.A catalogue of table definitions — names, columns and types — that query engines read so they know how to interpret files in S3.
Amazon AthenaThe query layer, inside a workgroup carrying a hard per-query scan ceiling.Runs SQL straight against files in S3 with no database to run. You are billed for the bytes each query scans, so file layout is a cost decision.
NAT GatewayBoth a cost centre and a subject: attributing its charges is one of the week's findings.Lets a private subnet reach the internet outbound while staying unreachable inbound. It is billed per hour AND per gigabyte, which is what makes it the expensive component in most small builds.
CloudWatchAnomaly detection where normal is unknown, static thresholds where zero is the right answer.AWS's metrics and logs service. Most services publish numbers into it automatically, without being asked to.

Two fields that look interchangeable and are not. traffic_path describes the path a packet took within AWS; next_hop_interface_type names what the traffic actually went through. Only the second can attribute NAT gateway cost, and the first is the one that sounds right.

Partition projection computes partitions from the S3 path rather than recording them. For flow logs that matters: the alternative is a crawler you pay to rediscover a layout that is fixed and documented.

Architecture and Workflow

The internet 200 scanning IPs aws AWS Cloud VPC — the observed network exposed host public subnet every packet REJECTed — the denial is the signal generator private subnet NAT gateway $0.045/hr + per GB The record, and the query 1 Flow logs v11 · 32 fields S3 Parquet 2 Athena one query Record version 11 carries the instance tag and the next-hop family — so the bill has an owner. The NAT gateway is most of this build's cost, and nothing on the charge itself says who caused it. The flow log does.
Two instances, two shapes of traffic. The expensive one is the one with no name on the bill.
  1. A generator instance in a private subnet reaches the internet through the NAT gateway, and reaches S3 through a gateway endpoint. Same instance, two destinations, two very different prices.
  2. An exposed instance in the public subnet has a security group with no ingress rules at all. It is internet-reachable and nothing can connect to it.
  3. Those refused connections produce the REJECT records the detection queries analyse. Removing them would not harden anything — it would delete the signal.
  4. Flow logs capture to S3 in Parquet, Hive-partitioned by date.
  5. A Glue table with partition projection makes it queryable without a crawler.
  6. Athena queries it inside a workgroup with a per-query scan ceiling, so a careless SELECT * cannot run up a bill.
  7. A daily analyzer publishes metrics; CloudWatch alarms watch them.

Design decisions

S3, not CloudWatch Logs. Half the ingest cost, and the only route to Parquet. CloudWatch Logs is right only if you specifically need metric filters on individual flows.

A spend ceiling on the workgroup, not a convention. Athena has no default limit. The ceiling is enforced before anyone else touches the account.

Mixed alarm types, on purpose. Anomaly detection where normal is genuinely unknown; static thresholds where zero is the right answer. Using anomaly detection everywhere costs about 30× more and detects less.

Prerequisites

RequirementDetail
AWS accountPermission for VPC, EC2, S3, Glue, Athena, Lambda, CloudWatch
Terraform>= 1.10
HCP TerraformWorkspace with OIDC to AWS. No static keys
Instance accessSSM Session Manager — no SSH, no key pairs
Regionus-east-1

Check which availability zone you land in. data.aws_availability_zones with state = "available" also returns opted-in Local Zones, which carry a deliberately reduced subset of regional services. One wrong AZ can look like three unrelated failures — see Troubleshooting. Filter on zone type.

The NAT gateway is the cost. About $33/month if left running, roughly three quarters of the total, and it bills whether or not a single packet flows. On this build, forgetting is the risk — not the rate.

Implementation

A small VPC built specifically to produce identifiable traffic, flow logs to S3 in Parquet, a Glue table, an Athena workgroup with a ceiling, and a daily analyzer with alarms. The lab guide carries the full Terraform and every query.

week-14-vpc-flow-logs-intelligence/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the modules
|   +-- modules/network/    VPC, subnets, NAT, the two traffic generators
|   +-- modules/capture/    flow logs to S3, Parquet, Hive partitioning
|   +-- modules/catalog/    Glue table with partition projection
|   +-- modules/analyzer/   workgroup ceiling, queries, Lambda, alarms
|
+-- queries/                the detection and attribution SQL
+-- scripts/                teardown verification
Only the directories that matter. The full repository is on GitHub.

One configuration field halves the cost — delivering to S3 rather than CloudWatch Logs. It is set once and never mentioned again, which is exactly why it is worth saying out loud.

AWS Glue table details showing Parquet classification and 37 columns
37 columns: 32 data fields plus 5 partition keys. Nothing crawled this table — the definition states where the data is.

Validation and Evidence

Every failure mode in this build is silent: zero rows and a green "succeeded" look identical to a working system. So each check asks for a number, not a status.

TestExpectedActualStatus
Partitions resolveRows returned for todayReturnedPass
Both traffic paths visibleNAT and gateway-endpoint traffic distinguishableDistinguishable — via next_hop_interface_typePass
NAT cost attribution via traffic_pathRows naming NAT trafficZero rows; analyzer published nat_bytes = 0Fail — wrong field
REJECT records capturedRefused connections appearAppeared, at scalePass
Port-scan alarm firesNotification sentFired and deliveredPass
Scan ceiling enforcedAn oversized query is refusedRefused at workgroup levelPass
Athena results showing egress grouped by traffic path, including free gateway endpoint rows and billed NAT rows
Path 7 to S3 costs nothing and has an owner. Path 1 rows are NAT-bound and have an owner. Path 8 rows are the NAT's own egress — the same bytes, with no team attached.
Athena results listing source IPs with reject counts and the ports they probed
Real unsolicited inbound: SSH and telnet sweeps, and probes at 8008, 8161, 9090. The top row with 36 distinct ports is my own generated test signal, included so the port-scan query had something deterministic to catch.
SNS notification email for the port-scan-detected alarm entering ALARM state
The detection path, end to end: flow log → Athena → custom metric → alarm → SNS → inbox. Account IDs redacted.

Security and Cost

Security

One deliberate exception runs through this build: the exposed instance is internet-reachable on purpose. Its security group has no ingress rules at all, so nothing can connect — and those denials are what generate the REJECT records the detection queries analyse. It holds no data and was destroyed with the rest of the stack.

No SSH, no key pairs, no inbound ports. Instance access is SSM Session Manager only. IMDSv2 required on both instances, encrypted root volumes.

Bucket hardening: all public access blocked, BucketOwnerEnforced, SSE-S3 with bucket keys, and a policy denying any non-TLS request.

Confused-deputy protection: the log delivery statement is scoped by both aws:SourceAccount and aws:SourceArn, so another account cannot point its flow logs at this bucket.

Least privilege. The analyzer's Athena permissions are scoped to the one workgroup carrying the scan ceiling — a broad grant would let it bypass the guardrail the design depends on.

Cost

These are rates and an estimate, not a measurement. This post was published before the charges posted and no Cost Explorer figure was recorded afterwards, so it says which it is rather than letting arithmetic read as a bill.

ItemRate48-hour build
NAT Gateway$0.045/hr + $0.045/GB~$2.16
2 × t4g.nano$0.0042/hr each~$0.40
Flow log ingest to S3$0.25/GB< $0.10
Athena$5/TB scannedcents
Anomaly alarms × 2$0.30/month each~$0.02

Left running: about $33 a month, roughly three quarters of it NAT gateway. The NAT bills whether or not a single packet flows and produces no usage signal to remind you it exists.

The cost controls are in the code rather than in advice: a 30-day expiry on raw logs, a noncurrent-version expiry alongside it (versioning is on, so expiry alone would create delete markers while the billed bytes stayed), and a multipart-upload abort for orphaned partial writes.

Troubleshooting and Lessons

One wrong availability zone, wearing three disguises

Problem. The first apply failed with three unrelated-looking errors at once: NatGateway NotAvailableInZone, gp3 VolumeTypeNotAvailableInZone, and t4g.nano not offered.
Root cause. One cause. data.aws_availability_zones with state = "available" also returns opted-in Local Zones. This account has the Dallas Local Zone enabled, so names[0] resolved to us-east-1-dfw-1a, and Local Zones carry a deliberately reduced subset of regional services.
Resolution. Filter on zone type and intersect with the instance type's actual offering list. Worth keeping the second half even with the filter: among genuine AZs, us-east-1e offers neither Graviton nor gp3.
Lesson. Three errors in three services, one cause. Check which AZ you landed in before debugging any of them.

traffic_path cannot attribute NAT cost

Problem. The NAT attribution query returned zero rows and the analyzer published a permanent nat_bytes = 0. Both looked like a broken pipeline.
Root cause. traffic_path describes the path a packet took within AWS. It is not the field that names what traffic went through.
Resolution. next_hop_interface_type.
Lesson. The field that looks right is measuring something else — and a wrong field returns zero rows, not an error.

Every failure here is silent

Problem. A broken table, a wrong field and an empty partition all present identically: zero rows, query succeeded.
Root cause. Athena reports on the query, not on whether the answer is plausible.
Resolution. Verify against delivered data rather than against your configuration — count rows for a window you know had traffic.
Lesson. No alarm fires for "your logs are unqueryable."

Cleanup and Key Takeaways

The workspace is VCS-connected, which blocks terraform destroy from a CLI checkout but not a destroy queued through HCP. The verification script does not destroy anything; it checks what survived, and distinguishes three outcomes rather than two — gone, still present, and could not verify. An expired session returning an empty result otherwise reads as a clean teardown while having checked nothing.

What tends to linger, and why:

  • The NAT gateway — the expensive one, with no usage signal to remind you it exists.
  • The Elastic IP — billed hourly once detached, and it outlives the NAT gateway it belonged to.
  • Anomaly detectors — a separate resource that survives a careless destroy.

Key takeaways

  • Pick S3, not CloudWatch Logs, unless you specifically need metric filters on individual flows. Half the ingest cost, and the only path to Parquet.
  • Verify against delivered data, not against your config. Every failure mode here is silent — zero rows and a green "succeeded" look identical to a working system.
  • Do not filter on traffic_path to find NAT cost. Use next_hop_interface_type.
  • Put a scan ceiling on the workgroup before anyone else touches it. Athena has no default spend limit.
  • Check which AZ you landed in. data.aws_availability_zones returns Local Zones too, and one wrong AZ can look like three unrelated failures.

References

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent