The query that never touches the firewall

Business Challenge

Two posts ago the finding was that a VPC endpoint changes the path a request takes and not the destination at the end of it. Last time it was that an egress domain allowlist matches a name the client itself supplies, scoped by a variable that defaults to inspecting nothing in a hub-and-spoke deployment. This one closes the set, and it is the least comfortable of the three: there is a path that neither control is on.

The account is well built by the standards of the previous two posts. Gateway and interface endpoints carry S3 and DynamoDB traffic, endpoint policies are scoped with aws:ResourceOrgID, and every route to the internet goes through a central inspection VPC where Network Firewall holds a forty-entry domain allowlist with HOME_NET correctly set to the spoke ranges. An application host is compromised. Data leaves anyway, steadily, for days.

It leaves as DNS. The process encodes a chunk of a file into the labels of a name and looks up <base32-of-the-data>.exfil.example.com. The VPC Resolver does what resolvers do and walks the delegation to whichever nameserver is authoritative for exfil.example.com, which is the attacker's. The data is in the question. The answer is irrelevant and is usually discarded.

AWS describes the technique in one sentence that explains why the other controls are silent: DNS tunneling “is used by attackers to exfiltrate data from the client by using the DNS tunnel without making a network connection to the client.” There is no TCP session from the instance to the attacker. There is nothing for a domain allowlist to match, because there is no TLS handshake and no HTTP request.

1The egress firewall is not on this path

This is not an oversight in the configuration. It is documented behaviour. AWS states that Network Firewall “provides filtering for both network and application layer traffic, but does not have visibility into queries made by Route 53 VPC Resolver.”

The Resolver is AWS infrastructure reached at a link-local address inside the VPC. Queries to it are not routed to the transit gateway, so they never reach the inspection VPC, so no rule in it applies. A firewall cannot filter traffic that does not pass through it, and a diagram showing all egress flowing through an inspection VPC is telling the truth about every path except this one.

Fix

Filter DNS where DNS is: a Resolver DNS Firewall rule group associated with the VPC.

2A domain list cannot enumerate this threat

The instinct is to add exfil.example.com to a block list. That works exactly once and only in hindsight. The base domain is chosen by the attacker, is registered for the purpose, and is unknown to you at the time it matters — and the traffic that gives it away is not the name but the shape: high query volume to a single zone, long labels, high entropy, answers nobody reads.

That is what DNS Firewall Advanced rules are for, and they cover three patterns a list cannot: Domain Generation Algorithms, DNS tunneling, and Dictionary DGA. The algorithms are managed and updated by AWS.

Fix

Treat the domain list and the Advanced protections as answering different questions. One knows names; the other knows behaviour.

3An allow rule above the threat lists switches them off

DNS Firewall evaluates rule groups by association priority from the lowest number up, and rules within a group the same way. The first match terminates evaluation. An Allow rule for a broad internal suffix, given a low priority number because someone was unblocking a legitimate service under pressure, ends inspection for everything it matches — including the managed malware and botnet lists sitting below it.

This is the same shape as the ordering trap in the previous post, where an Allow domain rule group's implicit drop pre-empted the Alert group behind it. Both are cases where a rule added to permit something quietly stops something else from ever running.

Fix

Keep the managed threat lists at the lowest priority numbers. Narrow allow rules go below them, not above.

Architecture

DNS Firewall is a feature of the VPC Resolver rather than a box in the data path, and almost everything that surprises people about it follows from that one fact.

Diagram: the three egress paths out of a VPC and which control is on each. The data path to S3 through a VPC endpoint is governed by an endpoint policy. The data path to the internet through a transit gateway and inspection VPC is governed by Network Firewall's domain allowlist. The DNS path to the Route 53 VPC Resolver is on neither, because AWS states Network Firewall has no visibility into queries made by the VPC Resolver, and it is filtered instead by Resolver DNS Firewall rule groups associated with the VPC. Below, the DNS Firewall evaluation order: rule groups by association priority from the lowest number up, rules within a group the same way, first match terminating evaluation, and no match resolving as normal, with the VPC fail mode defaulting to closed and returning SERVFAIL. — Three egress paths, three different controls, and one of them is not where the diagram usually puts it. DNS Firewall sits at the Resolver, so rule group order is the whole of its logic: first match wins and evaluation stops.

What it filters, and what it refuses to pretend to be

The scope statement is unusually blunt for AWS documentation, and it is worth reading as a limitation rather than a feature: “DNS Firewall only filters on the domain name. It does not resolve that name to an IP address to be blocked. Additionally, DNS Firewall filters DNS traffic, but it doesn't filter other application layer protocols, such as HTTPS, SSH, TLS, FTP, and so on.”

So blocking a name does not block the address behind it. A process holding a cached answer, or an address hard-coded, or a resolver of its own pointed at a public server on port 53 or 853, is not touched by any of this. DNS Firewall closes the resolution channel; it does not close the network. That is why it belongs alongside the controls from the previous two posts rather than instead of them.

Evaluation order is the entire logic

1

Rule groups, by association priority

Lowest numeric priority first. A VPC can carry several; they are tried in order until something matches.

2

Rules within a group, by rule priority

Same rule, one level down. Each rule holds exactly one domain list or one Advanced protection.

3

First match terminates

DNS Firewall stops evaluating and answers the Resolver. Everything below the matching rule is not consulted.

4

No match resolves normally

Exhausting every rule group is an implicit allow. A deny-by-default posture has to be written as an explicit block rule at the bottom.

What a block actually returns

A Block rule does not simply drop the query. It replies, and the reply you choose changes how the client behaves:

NXDOMAIN

Tells the client the name does not exist. Clean, and indistinguishable from a typo — which is either useful or misleading depending on whether you want the blocked party to know.

NODATA

Says the query succeeded but there is nothing to return. Some resolvers and libraries treat this very differently from NXDOMAIN on retry.

OVERRIDE

Returns a record you choose. The type must be CNAME, and the TTL defaults to zero, so nothing is cached unless you raise it. This is how you point blocked traffic at a sinkhole or a walled-garden page.

Alert, before any of them

AWS recommends creating a blocking rule as Alert first and counting what it would have caught. On a managed threat list that is not caution, it is the only way to size the false-positive cost.

Why This Architecture Holds Up

The failure mode is the right way round by default

DNS Firewall defaults to failing closed: if VPC Resolver receives no reply from DNS Firewall it blocks the query and returns SERVFAIL, which AWS describes as favouring security over availability. Fail open is available and does the opposite.

This is worth knowing precisely, because the default is the opposite of the intuition most people carry from other filtering products, and because the consequence is real: an impairment in DNS Firewall becomes a DNS outage for every VPC associated with a rule group, not a silent bypass. If you would rather have the bypass, that is a decision to take deliberately and write down — not one to discover during an incident.

One caution on the documentation

The overview page's high-level steps describe the fail setting the other way round, telling you to change the VPC configuration "if you want Route 53 VPC Resolver to block queries when DNS Firewall fails to send a response back". The dedicated VPC configuration page states the default plainly as closed, with a SERVFAIL response. Both pages are linked below; the configuration page is the one that matches the API, where the flag is FirewallFailOpen.

The managed lists are a real asset and an opaque one

AWS maintains lists for malware, botnet command and control, an aggregate list that contains all the others, and one sourced from GuardDuty's own detections. The data comes from AWS internal sources and Recorded Future and is updated continually. Using them is free.

They also cannot be downloaded or browsed, deliberately, so that attackers cannot design around a published list. That is a sound trade and it has an operational consequence: when a list blocks something you needed, you cannot diff it or audit it. The remedy AWS documents is to find the block in the Resolver query log, then add a narrow allow rule for that one name at a lower priority number than the managed rule — the one legitimate case for putting an allow rule above a threat list, and it should be one name wide.

Why the query log is not the flow log

A VPC flow log records addresses, ports and byte counts. It cannot record a name, because by the time a packet exists the name is gone. The Resolver query log is the only place the requested domain appears, and it is also where the rule group, the rule action and the matching managed list are recorded — which is what makes a false positive diagnosable at all.

Key Architecture Decisions

Decision	Choice	Reasoning
Where DNS is filtered	Resolver DNS Firewall, not the egress firewall	Network Firewall has no visibility into VPC Resolver queries. This is documented behaviour, not a gap in the rules.
Rule group order	Managed threat lists at the lowest priority numbers	First match terminates evaluation, so anything above the threat lists can switch them off for the names it matches.
Tunneling and DGA	Advanced protections, alert first, then block	These are behavioural patterns a domain list cannot enumerate, because the attacker chooses the domain and you learn it afterwards.
Confidence threshold	Start High, move down with evidence	High detects only well corroborated threats with few false positives; Low catches the most and costs the most to triage.
Fail mode	Leave it closed unless you have decided otherwise	The default favours security. Fail open turns an impairment into a silent bypass rather than a visible outage.
Query logging	On, and retained	It is the only record of the name, and the only way to diagnose a managed-list false positive.

Two settings that decide more than they appear to

Domain redirection. By default a rule inspects every domain in a redirection chain — the CNAME and DNAME targets, not just the name that was asked for. You can tell it to trust the rest of the chain after the first name, which is faster and weaker. The trust applies only within a single query transaction: if a client separately queries a redirection target directly, that is evaluated as an independent query with no inherited trust.

Query type. A rule can be scoped to one DNS query type, and if you do not choose one it applies to all of them. This is how you block a domain generally while still permitting MX. It is also worth remembering when reasoning about tunneling, which does not need A records and is perfectly happy in TXT or NULL.

Advanced protections cannot build a walled garden

The Allow action is available only on rules that carry a domain list. In an Advanced rule you can block or alert and nothing else. So a default-deny posture is always built from domain lists, and the tunneling and DGA protections sit alongside it as detection — they are not an allowlisting mechanism and cannot be made into one.

Closing Thought

Three posts, three controls, one question underneath all of them: which path does this actually sit on? An endpoint policy sits on the path to a service and says nothing about the internet. A domain allowlist sits on the path to the internet and reads a field the client wrote. DNS Firewall sits on the resolution path and never sees a packet.

None of them is weak. What is weak is the sentence in the architecture review that says “egress is controlled”, because that sentence has no path in it. The version worth writing names one: traffic to AWS services is constrained by endpoint and organisation policy; traffic to the internet is constrained by a domain allowlist that matches a client-supplied name; DNS resolution is constrained by rule groups on the Resolver. Three clauses, because there are three paths, and the value of writing it that way is that the missing fourth one becomes visible the moment somebody adds it.

For this series that has been the recurring lesson rather than any particular service: a control is only as good as the honesty of the sentence describing what it observes.

Next in this series

Networking & Security — the last mile of egress: what a VPC flow log can and cannot answer during an investigation, why the fields you want are the ones most often left at default, and how Traffic Mirroring changes the question when a flow log is not enough.

AWS Architecture Series #46 — The query that never touches the firewall