Week 5 — The Expensive Thing Should Be the Disposable Thing

Why — The Problem This Solves

Weeks 1 through 4 built governance and a vending pipeline. Every one of them was destroyed at the end of the day, and nothing was lost, because nothing depended on them.

The hub network is different. Spokes peer into it. Private endpoints resolve through its DNS estate. Egress leaves through its firewall. Six later weeks assume it exists — so "tear it all down" stops being the responsible default and becomes the thing that breaks next week.

But a hub with an Azure Firewall in it bills by the hour, permanently. Standard is roughly $967/month if left running. That is a real number for a lab, and "just remember to delete it" is not a design.

What You Need to Know — Skills & Tools

• Hub and spoke topology — and that a peering is two resources, one declared from each side.
• Azure Firewall SKUs, and specifically that Basic is not simply a cheaper Standard — it has a structural requirement Standard does not.
• Firewall Policy hierarchy — a parent policy a landing zone cannot edit, and a child it can.
• Private DNS zones for private endpoints — where they should live, and why per-spoke zones fail silently.
• User-defined routes — the thing that makes a firewall load-bearing rather than merely present.
• Terraform provider aliases, for a configuration spanning two subscriptions.

Architecture — How It Fits Together

The hub lives in the connectivity subscription and the spoke in the landing zone subscription — two subscriptions, two provider aliases, one configuration. That is what the management group hierarchy built in the bootstrap is for, and this is the first week that proves the split earns its keep: the two halves can be owned by different teams, on different lifecycles.

How We Built It — In Deployment Order

1. The hub, the spoke, and the peering between them

A peering is two resources, not one. Each side is declared independently, and traffic only crosses when both exist. One side alone sits in Initiated — a resource that exists, reports healthy in the portal, and moves no packets.

That is why the check reads the state rather than the existence:

2. The private DNS estate

Six privatelink zones, created in the hub and linked to both virtual networks. The names cannot be invented — each Azure service publishes the exact zone its private endpoints register into, and a zone one character off resolves nothing while looking entirely correct in the portal.

They live in the hub because a private endpoint must resolve to the same record everywhere. Per-spoke zones are the classic mistake: resolution works in the spoke that owns the endpoint, and silently returns the public IP everywhere else. That failure surfaces much later, in some unrelated week, and looks exactly like a firewall problem.

Everything to this point costs approximately nothing and is designed to stay up. The next step is where the meter starts.

3. The firewall — and what Basic quietly requires

Read that blade carefully, because it contains the week's first real finding. There is a Management subnet and a Management public IP alongside the ordinary ones. Those are not optional:

Firewall Basic has a mandatory requirement to be configured with a management NIC.

Basic requires a second subnet named AzureFirewallManagementSubnet and a second public IP, neither of which Standard needs. That is a virtual network design decision, not a pricing toggle — so this hub creates the management subnet unconditionally, at every tier. Changing SKU later is then a variable change, rather than re-addressing a hub that six weeks are already peered into. An empty /26 costs nothing.

Basic's other limits are worth knowing before choosing it: no DNS proxy ("uses Azure DNS only"), no network-level FQDN filtering, no web categories, threat intelligence in alert mode only, and a 250 Mbps ceiling. Private endpoint resolution still works — the zones above are linked per-VNet and resolve through Azure DNS — but the firewall cannot be the central resolver for spokes, which is the pattern most reference architectures show.

4. The policy hierarchy

A parent policy holds what every landing zone gets and cannot remove. A child inherits it and adds its own. Child rule collection groups are evaluated after the parent's, so the parent always wins — which is the entire reason to run two rather than one.

Two details here. The policy tier must match the firewall tier — a Standard policy on a Basic firewall is refused. And the portal states the trade-off plainly: TLS inspection (Premium): Not supported with basic policy.

The inheritance is easiest to verify from the firewall blade in step 3, which reports 1 rule in 1 collection and 1 rule in 1 collection (inherited from base policy). That second line is the hierarchy working, read from the deployed firewall rather than from the template that claimed it.

5. The route that makes it load-bearing

Without this step, the firewall exists, bills, and carries nothing. The spoke's default route has to point at it.

0.0.0.0/0 → VirtualAppliance → 10.0.0.4. That address is the firewall's private IP from step 3 — worth comparing the two screenshots, because a route pointing at the wrong address is indistinguishable from a correct one until traffic stops.

The route is gated on the same variable as the firewall, deliberately. A 0.0.0.0/0 route whose next hop no longer exists does not fail loudly; it blackholes the subnet.

Validation with everything up: 5 passed, 0 failed. Without the firewall, the same script reports 4 passed and skips the route check — the absence of that route is the correct state, not a gap.

Challenges — What Actually Went Wrong

The teardown failed, and the script reported success

This is the one worth the whole post. Cleanup died with:

FirewallPolicyUpdateFailed - Put on Firewall Policy afwp-lz-dev...
Failed with 1 faulted referenced firewalls

Terraform tried to update the child policy while the firewall still referenced it. The destroy aborted. The firewall stayed up and kept billing — and the script still exited 0, so nothing signalled a problem.

And the safety check could not run at all

The script was supposed to catch exactly that. It verified with az network firewall list — which requires the azure-firewall extension and, in a non-interactive shell, dies on the dynamic-install prompt:

EOFError: EOF when reading a line

So the check whose entire purpose was to detect a surviving firewall was itself incapable of running. Two independent failures lined up: an apply that failed silently, and a verification that could not execute. Either alone would have been caught by the other.

Recovery was az resource delete --ids <firewall> — which needs no extension — followed by a re-apply. Eight minutes. The script now checks the apply's exit code explicitly and verifies with az resource list.

A query that returned null instead of failing

az network vnet subnet show --query addressPrefix now returns null: the value moved to addressPrefixes[]. The old field is still in the response, so the query succeeds and returns empty — which is indistinguishable from "the subnet does not exist". Both firewall subnets reported as missing when they existed and were correctly sized.

An azurerm 5.x schema change

azurerm_private_dns_zone_virtual_network_link takes private_dns_zone_id in 5.x. In 4.x it took resource_group_name plus private_dns_zone_name, so copying a link block from almost any published example fails validate on both arguments at once.

Security — Controls at Every Layer

• Egress is forced through the firewall by the spoke's default route, not left to per-resource configuration. A workload cannot opt out of it.
• The parent policy cannot be edited by the landing zone. Platform rules are inherited and evaluated first; a landing zone team can add rules and cannot remove the baseline.
• Private DNS zones live in the hub, so private endpoint resolution is consistent across every spoke rather than per-team and divergent.
• Auto-registration is off on every zone link. These zones exist for private endpoints, which create their own records; auto-registration would additionally register every VM NIC in the linked network into a privatelink zone.
• Basic's threat intelligence is alert-only — it cannot deny. Worth stating explicitly, because "threat intelligence is enabled" reads as protection and on this tier it is telemetry.

Cost — And What It Really Came To

Prices read from the Azure retail price API for South Central US on the day:

The plan was to run the firewall for about an hour — roughly $0.41. It actually ran 1 hour 47 minutes, about $0.72.

Nothing about Azure caused the overrun. The firewall finished deploying, the screenshots were taken, and then it sat there while attention was elsewhere. The hourly rate was never the risk. An absence of a deadline was — which is precisely the failure mode the two-layer split exists to contain, arriving in the one window where the split does not protect you.

Cleanup

Cleanup removes the firewall, both public IPs and both policies by re-applying with the flag off — not by destroy -target, which would leave the route pointing at an address that no longer answers. Both are gated on the same variable, so they go together in dependency order.

The public IPs matter more than they look. A standard static IP left behind is $0.005/hour: trivial per hour, permanent if nobody checks. The script now verifies that zero firewalls and zero public IPs remain before it reports clean.

The hub, the spoke, the peerings and the DNS estate stay up. That is the point.

References

• Azure Firewall features by SKU
• Deploy and configure Azure Firewall Basic
• Private endpoint DNS configuration
• Hub-spoke network topology in Azure
• Week 5 code in the lab repository

Key Takeaways

• Split a week by what it costs, not by what it does. When one resource carries the entire hourly bill and everything else is free, those are two different lifecycles wearing one name.
• Basic is not a cheaper Standard. It mandates a management subnet and a second public IP, and it has no DNS proxy. Decide the tier before addressing the hub, not after.
• A peering is two resources. One side alone exists, reports healthy, and carries nothing — so check the state, not the existence.
• Put private DNS zones in the hub. Per-spoke zones resolve correctly where the endpoint lives and return the public IP everywhere else.
• A cleanup script that cannot fail loudly is not a cleanup script. Mine exited 0 while the firewall kept billing, and its safety check could not run in the shell it ran in. Check exit codes, and make sure the verification command actually works where it executes.