Business Challenge
Both of today's zone types are private zones, both are created by the same command with a different flag, and both are empty by design. A forwarding zone is a type of Cloud DNS managed private zone that forwards requests for that zone to the IP addresses of its forwarding targets; a peering zone is a type of Cloud DNS managed private zone that follows the name resolution order of another VPC network.
The resemblance is the problem. They solve different halves of hybrid resolution, they fail in different ways, and the way forwarding fails is specifically designed to defeat the test you would reach for.
This is the documented headline behaviour: Cloud DNS cannot use transitive routing to connect to a forwarding target. The page walks a setup where on-premises is joined to vpc-net-a, vpc-net-a is peered to vpc-net-b with custom routes exported and imported, and the forwarding zone is authorized for vpc-net-b — then gives you two tests to run from the same VM.
Query the metadata server for the forwarded name: this query fails (expectedly) because Cloud DNS does not support transitive routing to forwarding targets. Query the target name server directly for the same name: although Cloud DNS does not use this path, this query demonstrates that transitive connectivity succeeds.
Correct approach
Stop looking for a route or a firewall rule, because both are already correct — the second test proves it. You can use a Cloud DNS peering zone to fix this invalid scenario: peer from vpc-net-b to vpc-net-a, and authorize the forwarding zone for vpc-net-a, which is the network that can actually reach the target. The documentation notes you can perform these steps in any order.
It does not, and this is the single most useful sentence in the Cloud DNS documentation for anyone who read #81: DNS peering and VPC Network Peering are different services. Then, explicitly: VPC Network Peering doesn't automatically share DNS information.
Correct approachTreat them as unrelated, in both directions. VPC Network Peering exchanges subnet routes and nothing else — names were never part of that deal. And the converse holds too: DNS peering can be used with VPC Network Peering, but VPC Network Peering is not required for DNS peering, and peering zones don't depend on VPC Network Peering. You can have either without the other, and most confusion here comes from assuming one implies the other.
Usually the return path, which is the requirement most easily satisfied by accident in a simple network and most easily missed in a real one. Forwarded queries do not come from your VM — by default, queries forwarded to private targets originate from 35.199.192.0/19, and Cloud DNS uses the 35.199.192.0/19 source range for all customers.
Correct approachVerify the reply has a way home, and specifically the right way home: your on-premises network must have a route for the 35.199.192.0/19 destination, whose next hop is in the same VPC network where the request originated. The constraint is sharper than "a route exists" — Cloud DNS cannot send responses from Type 2 forwarding targets over the internet or through a different VPC network. A second tunnel into a different VPC network does not count, however healthy it looks.
Architecture
Forwarding reaches name servers. Peering reaches resolution orders. Only one of them crosses a VPC boundary.
Neither zone accepts a record
Worth stating first because it reframes both features. You cannot add records to a forwarding zone directly; the data comes from one or more configured target name servers. And you cannot add records to a peering zone directly; the data comes from the producer VPC network according to its VPC name resolution order.
So what you are creating is not a zone in the ordinary sense. It is an entry in the resolution order from #98 — a claim over a suffix, with a delegation of the answer to somewhere else. Everything #98 established still applies: the authorized-network list is the whole of visibility, matching is by longest suffix, and a zone that wins the match and cannot answer returns NXDOMAIN rather than falling through.
Forwarding: four conditions, three of them outside Cloud DNS
A forwarding zone is the only thing in this block that needs you to configure something on a device Google does not operate, which is why it accounts for a disproportionate share of hybrid DNS tickets.
The firewall. For Type 1 targets, create an ingress allow firewall rule for TCP and UDP port 53 traffic, applicable to your forwarding targets in each authorized VPC network. On-premises needs the same — for Type 2 targets, configure an on-premises network firewall and similar equipment to permit TCP and UDP port 53. Both protocols. A UDP-only rule works until a response needs TCP, which is to say until DNSSEC or a large record set appears.
The route out. A subnet route for a target inside the VPC network, or a custom dynamic or static route for one on-premises — with one exclusion worth noting, since tagged static routes are invisible to Cloud DNS.
The route back. Covered above, and the one that breaks real estates: the reply must return through the same VPC network the query left from.
The reply must come from the target itself. This is the condition nobody anticipates. Cloud DNS requires that the forwarding target that receives packets be the one that sends replies to 35.199.192.0/19. Specifically: if your forwarding target sends the request to a different name server, and that other name server responds to 35.199.192.0/19, Cloud DNS ignores the response. The reason is stated — for security reasons, Google Cloud expects the source address of each target name server's DNS reply to match the IP address of the forwarding target.
A great many corporate DNS deployments are a load-balanced front end that hands queries to a pool of resolvers, and a pool member replying directly is normal, efficient behaviour everywhere else. Here it is silently discarded, because the source address of the reply does not match the forwarding target you configured.
The symptom is a timeout followed by SERVFAIL, not a refusal, so it reads as unreachability — and a packet capture on the on-premises side shows the query arriving and a reply being sent, which makes the Google side look like the problem. Point the forwarding target at whatever address actually answers.
Forwarding: choosing targets, and what happens when they stop answering
Multiple targets are ranked, not round-robined. When you configure two or more forwarding targets, Cloud DNS uses an internal algorithm to select a forwarding target, and the two inputs are the ones you would hope for: the higher the number of successful DNS responses processed by the forwarding target, and the lower the latency (round-trip time) for communicating with the forwarding target. When nothing answers at all, Cloud DNS synthesizes a SERVFAIL response.
The caching behaviour is the part worth designing around. Normally Cloud DNS maintains a cache of responses from reachable forwarding targets for the shorter of the following time spans — 60 seconds, or the record's own TTL. But the failure case is deliberately generous: when all of the forwarding targets for a forwarding zone become unreachable, Cloud DNS maintains its cache of the previously requested records in that zone for the duration of each record's TTL.
That is a real availability property and it is worth knowing which way it cuts. A long TTL on your on-premises records buys you a grace period when the tunnel drops; a short one means resolution fails almost immediately. If hybrid DNS is on your critical path, the TTL on the records you forward to is an availability decision, made on a server Google does not manage.
One asymmetry in the target configuration catches people building from Terraform: when using an FQDN as a forwarding target, you can only use one, where IP targets go up to number of forwarding targets in a forwarding zone 50. An FQDN target does resolve wide — the forwarding target can resolve to up to 50 IP addresses — but it is one entry, and you can't use an FQDN as an alternative name server in server policies at all, which is #100's territory.
Peering: borrowing a resolution order
A peering zone does something more sweeping than its name suggests, and the vocabulary matters. The VPC network where the DNS peering zone performs lookups is called the DNS producer network, and the VPC network that is authorized to use the peering zone is called the DNS consumer network.
What the consumer gets is not a zone. Lookups for records in the peering zone's namespace follow the DNS producer network's name resolution order — the whole ordered list from #98, as evaluated in the producer network: its private zones, its forwarding zones, its Compute Engine internal names, its alternative name server. The resolution-order page says it exactly: if the most specific match for the query is the name of a VPC network-scoped peering zone, Cloud DNS stops the current name resolution process and begins a new name resolution process from the perspective of the peering zone's target VPC network.
Two properties follow. DNS peering is a one-way relationship — the producer gains nothing, and bidirectional resolution needs two zones in two projects. And the transitivity is bounded but non-zero: transitive DNS peering is supported, but only through a single transitive hop, meaning no more than three VPC networks (with the network in the middle being the transitive hop) can be involved.
That is the sharpest contrast with #81 available anywhere in this block. VPC Network Peering is non-transitive, full stop. DNS peering is transitive exactly once. Same word, different service, different rule.
Creating a peering zone requires a grant in a project you may not administer: you must be signed in to the Google Cloud console as an Identity and Access Management (IAM) member who has the DNS Peer role (roles/dns.peer) to the project that contains the producer VPC network, backed by the permission dns.networks.targetWithPeeringZone to create a peering zone.
That is deliberate and it is the right shape. A peering zone exposes the producer network's entire resolution order to the consumer, so the producer's owner has to agree — which also means a consumer team cannot build this alone, and the request belongs in the design conversation rather than at the end of it. The relationship can cross organisations: while DNS producer and consumer networks are typically part of the same organization, cross-organizational DNS peering is also supported.
Why This Architecture Holds Up
The asymmetry looks arbitrary until you ask where the packet goes in each case, at which point it becomes the only possible answer.
A forwarding zone produces a packet on the network. Cloud DNS sends a real UDP or TCP query from 35.199.192.0/19 — a /19, so 8,192 addresses, shared by every Google Cloud customer — to an IP address, and that packet needs a route out, a firewall that permits it, and a route home, all of which exist in exactly one VPC network. The range is not internet-routable: this range is only accessible from a Google Cloud VPC network or from an on-premises network connected to a VPC network. So a forwarding zone's reach is the reach of one network's routing table, and asking it to traverse a peering to a second network asks a packet to come home down a path that does not exist.
A peering zone produces no packet at all. It restarts the resolution process in the producer network's context — a control-plane operation inside Cloud DNS, with no source address and no return route to arrange. Which is why it can cross a boundary that forwarding cannot, why it needs no VPC Network Peering underneath, and why its limit is a hop count chosen by Google rather than a routing constraint.
Forwarding is data plane. Peering is control plane. That one line predicts every difference in this post.
Forwarding needs firewall rules, routes, a return path and a matching source address, because it puts a packet on a wire. Peering needs an IAM grant, because it changes who may ask Cloud DNS a question. Forwarding stops at the VPC boundary; peering is the thing that crosses it. And when a query must cross a boundary and then reach a name server, you need both — peer to the network that can forward, and forward from there. Which is precisely the fix the documentation gives for its own invalid scenario.
Choosing between them, stated once
If the answer lives on a name server — on-premises BIND, Active Directory, a third-party private resolver — forward to it. If the answer lives in another VPC network's resolution order — its private zones, or a forwarding zone it already owns — peer to it. The most common real design is both at once, and in that case the forwarding zone belongs in the network that has the connectivity, not the network that has the question.
Key Architecture Decisions
| Situation | Decision | Because |
|---|---|---|
| The answer is on a name server you run | Forwarding zone | Its data comes from configured target name servers. |
| The answer is in another VPC network's resolution order | Peering zone | Its data comes from the producer network's own order. |
| Forwarding target reachable from the VM, resolution still failing | Add a peering zone; move the forwarding zone | Forwarding cannot use transitive routing, by design. |
| You have VPC Network Peering and expect DNS to follow | It does not; configure DNS peering separately | VPC Network Peering doesn't automatically share DNS information. |
| Considering VPC Network Peering just to enable DNS peering | Unnecessary | Peering zones don't depend on VPC Network Peering. |
| Writing the firewall rule for forwarding | Allow TCP and UDP port 53 from 35.199.192.0/19 | Both protocols are documented as required. |
| Reply path for an on-premises target | Route 35.199.192.0/19 back through the same VPC network | Responses cannot return via the internet or another VPC. |
| Corporate DNS behind a load balancer or resolver pool | Target the address that actually answers | A reply from a different name server is ignored. |
| Using a tagged static route to reach the target | Use an untagged custom route instead | Tagged static routes are excluded for forwarding. |
| Hybrid DNS on the critical path | Treat the forwarded records' TTL as an availability setting | When all targets go unreachable, the cache holds for the TTL. |
| Wanting several FQDN forwarding targets | Not possible; use IP addresses | Only one FQDN target is allowed per zone. |
| Needing resolution in both directions | Two peering zones, one each way | DNS peering is a one-way relationship. |
| A chain of four VPC networks | Redesign — peer directly to the producer | Only one transitive hop, so at most three networks. |
| Planning a peering zone against another team's network | Get roles/dns.peer in their project first | The grant lives in the producer's project, not yours. |
| Forwarding to an external resolver on the internet | Standard routing only | Private routing to Type 3 targets is not supported. |
The one to look at today
For every forwarding zone you have, check two things that are not in Cloud DNS. First, that the on-premises or peer-network side has a route for 35.199.192.0/19 whose next hop is in the same VPC network the queries leave from — not merely that a route exists. Second, that the IP you configured as the forwarding target is the IP that sends replies, rather than a virtual address in front of a pool. Both are satisfied by accident in a simple network and both fail silently once the network grows.
Closing Thought
The thing I will carry out of this post is the two-test diagnostic, because it is a better piece of engineering writing than most documentation attempts. Rather than asserting that forwarding is non-transitive, the page hands you a pair of queries from the same VM whose results can only be explained by that fact: the resolver path fails, the direct path succeeds, and no amount of route inspection will reconcile them. It converts an architectural rule into something you can observe in thirty seconds.
It also quietly makes the general point. Connectivity and resolution are different systems with different rules, and a successful ping has never been evidence about DNS. This block has shown the same lesson from several angles — #91's one-way reachability, #96's ignored default route, #98's NXDOMAIN under a working public zone. Here it is at its cleanest: everything is reachable, nothing resolves, and both facts are correct.
#100 finishes the Cloud DNS block with server policies — inbound, which lets on-premises hosts query Cloud DNS, and outbound, which supersedes the entire resolution order this post has been standing on.
Comments