Home› Blog› AWS Architecture Series #47 — The log that records that it happened, not what…
AWS Architecture AWS Architecture Series

AWS Architecture Series #47 — The log that records that it happened, not what

An investigation asks which host reached an external address at 03:14. The account has flow logs on every VPC, enabled at the default format, and the answer is not in them: behind a NAT gateway every record shows the NAT gateway as the source. The field that would have answered it arrived in version 3 and had to be chosen before the traffic happened.

Sep 9, 2026 11 min read
Verified against current vendor documentation on 9 September 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

The last three posts were about controls that inspect a path. This one is about the record those controls leave behind, and it has the same shape as all of them: the artefact is real, it is correctly configured, and it answers a narrower question than the person reading it believes.

An incident review asks something simple. At 03:14 a connection was made to an address now known to be bad. Which host made it? The account is well instrumented by the usual standard: flow logs enabled on every VPC, delivered to a central S3 bucket in the log archive account, retained for a year. The query runs. Every matching record shows the same source address, and it is the NAT gateway.

That is not a bug, and it is not a gap in the deployment. It is what the default format records. The field that distinguishes the original sender from the intermediate hop is pkt-srcaddr, it arrived in version 3, and the default format is the version 2 fields — which AWS states you cannot customize or change. The only way to have had that field at 03:14 was to have chosen it before 03:14.

1Behind a NAT gateway, every host is the same host

AWS documents this directly as the purpose of the field: use pkt-srcaddr with srcaddr “to distinguish between the IP address of an intermediate layer through which traffic flows, and the original source IP address of the traffic”, and the first example it gives is a NAT gateway.

The second example is worse for anyone running containers: an EKS pod's IP address differs from the network interface of the node it runs on. Without the packet-level fields, a cluster of two hundred pods produces flow records that all name a handful of nodes.

Fix

Custom format, with pkt-srcaddr and pkt-dstaddr. There is no retrofit.

2It is telemetry, and AWS says so

There is a sentence in the field reference that deserves more attention than it gets: “Metadata fields that do not come directly from the packet header are best effort approximations, and their values might be missing or inaccurate.”

Alongside it, log-status has a value called SKIPDATA, meaning records were skipped during the aggregation interval “because of an internal capacity constraint, or an internal error”. And the timestamps carry up to sixty seconds of slack in either direction, by design.

None of that makes flow logs useless. It makes them evidence of the kind that supports a hypothesis rather than the kind that settles one, and an investigation that treats absence of a record as proof of absence of a flow has misread what it is holding.

Fix

Corroborate. Treat a flow log as one source among CloudTrail, Resolver query logs and the appliance's own logs.

3The answer arrives after the incident is over

The maximum aggregation interval defaults to 10 minutes. Delivery to S3 then takes about 10 minutes more, on a best-effort basis, and may be later. So roughly twenty minutes can pass before a flow is readable — which is fine for an investigation and useless as a trigger.

The mitigation is partly free: on a Nitro-based instance the aggregation interval is always one minute or less regardless of the maximum you specified, so much of a modern fleet is already better than the setting suggests. Setting the maximum to 1 minute makes that explicit everywhere else, at the cost of a higher record volume.

Fix

Do not build alerting on flow log latency. Build it on the control planes that decide, and use flow logs to reconstruct.

Architecture

The available fields have grown to version 11. The default has not moved from version 2. Almost everything worth having is in the gap.

Diagram: what the default VPC flow log format records against what the later field versions add. On the left, the version 2 default: version, account, interface, source and destination address and port, protocol, packets, bytes, start, end, action and log status — enough to say that two addresses exchanged packets. On the right, the opt-in fields that answer investigation questions: pkt-srcaddr and pkt-dstaddr in version 3 which separate the original sender from an intermediate NAT gateway or EKS node, flow-direction and traffic-path in version 5, ECS task and container fields in version 7, reject-reason in version 8, encryption-status in version 10, and instance and interface tags plus next-hop fields in version 11. Across the bottom, the accuracy caveats: metadata fields not taken from the packet header are best-effort approximations, SKIPDATA means records were dropped for internal capacity reasons, timestamps carry up to sixty seconds of slack, and no version of a flow log contains payload — which is where Traffic Mirroring begins.
The default format is the version that answers the least, and it is the one that cannot be changed. Every field that turns "these addresses talked" into "this host did this" is opt-in and cannot be added after the fact.

The fields that change what a query can ask

pkt-srcaddr, pkt-dstaddr · v3

The original addresses rather than the intermediate hop. Without these, NAT gateways and EKS nodes absorb the identity of everything behind them.

flow-direction, traffic-path · v5

Ingress or egress, and which of eight egress paths the traffic took — including 8 for an internet gateway and 7 for a gateway VPC endpoint. This is the field that answers the question the last three posts kept asking.

reject-reason · v8

BPA for VPC Block Public Access, EC for a rejection by VPC encryption controls, - for everything else. It turns some REJECTs from a mystery into an attribution.

encryption-status · v10

Read the footnote before trusting it: for interface and gateway endpoints AWS “does not look at packet data to determine encryption status” and instead infers it from the port.

The paths the addresses do not reveal

traffic-path deserves singling out because it closes a loop this series has been running for four posts. A destination address tells you where a packet went. It does not tell you how it left, and in a hub-and-spoke account the difference between value 8, an internet gateway, and value 7, a gateway VPC endpoint, is the difference between traffic that passed the egress firewall and traffic that never went near it.

Two fields in version 11 extend the same idea: interface-type, which names the local interface as nat_gateway, transit_gateway, vpc_endpoint and so on, and the next-hop-* family, which names the interface, subnet, AZ and VPC on the other side. Together they let a query reconstruct a path rather than infer one.

What no version contains

There is no payload in a flow log at any version, and there never will be — the record is a per-flow aggregate, not a capture. When the question moves from who talked to whom to what was said, the answer is Traffic Mirroring, which copies traffic from an elastic network interface to an out-of-band appliance for content inspection, threat monitoring and troubleshooting.

It has one property worth knowing for an investigation with an adversary in it: the copy is taken at the network interface, “which cannot be disabled or tampered with from a user space”. An agent on a compromised host can be silenced. This cannot.

Why This Architecture Holds Up

The custom format costs almost nothing to choose well

A custom format lets you specify any number of the available fields, in any order, with a minimum of one. There is no per-field charge; the cost is volume, and volume is driven far more by the aggregation interval and how much traffic you log than by field count. The asymmetry is stark: a few extra fields cost a little storage, and their absence costs an entire investigation.

The decision is dated, and the date is before the incident

This is the property that makes flow log configuration different from most settings. Changing the format changes what is written from now on; it does nothing for the year of logs you already have. Every other control in this series can be tightened after you discover it was too loose. This one cannot be tightened backwards at all.

Permissions are part of the format

Some of the most useful fields are not purely a format choice. The ECS fields need permission to call ecs:ListClusters, and depending on the field also ecs:ListTasks, ecs:ListServices or ecs:ListTaskDefinitions. The tag fields need ec2:DescribeTags and iam:CreateServiceLinkedRole.

That matters operationally because a format specifying fields the flow log role cannot populate does not fail loudly — it writes a -. A column of dashes six months into retention is indistinguishable at a glance from traffic that had no ECS task behind it.

Key Architecture Decisions

Decision Choice Reasoning
Format Custom, never default The default is the version 2 fields and cannot be changed. Every field that identifies a host behind an intermediate hop arrived later.
Minimum useful additions pkt-srcaddr, pkt-dstaddr, flow-direction, traffic-path Who really sent it, which way it was going, and which egress path it took. These four turn a flow log into an investigative record.
Aggregation interval 1 minute where the volume is affordable Nitro instances are already at 1 minute or less regardless. Setting it makes the rest of the fleet consistent rather than assumed.
Role permissions Granted for every field in the format A field the role cannot populate is written as a dash, silently, and looks like a legitimate absence for as long as you keep it.
Payload Traffic Mirroring, scoped and time-boxed Flow logs never contain payload. Mirroring does, and is billed per session-hour until the session is deleted.

The Traffic Mirroring bill does not stop when the instance does

This is the operational trap in the feature, and AWS lists it plainly. Charging is hourly per active session, and it continues in all of the following: you detached the network interface from the mirror source, you stopped or terminated the mirror source, or you changed the source to an unsupported instance type. The session is the billable object, not the traffic.

So a mirroring session opened for a two-day investigation on an instance that was terminated the following week keeps billing until somebody deletes the session. Put the deletion in the investigation's closing checklist, not in a reminder.

Filters and truncation before volume

Traffic Mirroring supports filters and packet truncation so you can extract only the traffic of interest. Mirroring everything to a fleet behind a Network Load Balancer or a Gateway Load Balancer with a UDP listener is possible and almost never the right first move — the headers usually answer the question, and truncating to them is dramatically cheaper than carrying payloads you will not read.

Closing Thought

There is a pattern across these four posts that is worth naming now that the set is complete. An endpoint policy, an egress allowlist, a DNS firewall and a flow log are all real controls, all correctly configured in the accounts where they disappoint, and all quietly narrower than the sentence used to describe them. The endpoint changed the path, not the destination. The allowlist read a name the client wrote. The firewall was not on the DNS path. The log recorded the hop, not the host.

What separates the flow log from the other three is timing. The other controls can be tightened the day you understand them, and the improvement applies immediately. A flow log format applies only forward. The year of records you already hold was fixed by a decision somebody made when they enabled it — very often by accepting a default, which is not a decision at all.

So the practical instruction is narrower than usual and easier to act on: go and look at what format your flow logs are actually using. If the answer is the default, the fix costs an afternoon, and every day you wait is another day of records that cannot answer the question you will eventually ask them.

Next in this series

Cost & Governance — what all this logging costs once it is turned on properly: the difference between what you pay to ingest and what you pay to keep, why the log archive is usually the third-largest line on a security budget, and the retention decisions that are cheaper to take at the start than to unwind.

Topics: AWS Architecture AWS Architecture Series

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent