Business Challenge
#75, #76 and #77 all ended in the same place: the record is incomplete, and the incompleteness leans toward silence. Packet Mirroring is the escape from that. It clones the traffic of specified instances in your VPC network and forwards it for examination, and the scope is total — Packet Mirroring captures all traffic and packet data, including payloads and headers. Against #77 specifically: it exports all traffic, not only the traffic between sampling periods.
That solves the evidence problem and creates three new ones. The mirror is taken in a place that costs you bandwidth, it copies data in a state you may not be allowed to copy, and the product is on its way out in favour of a replacement whose adoption switches this one off.
They will. The mirroring happens on the virtual machine (VM) instances, not on the network. And therefore: consequently, Packet Mirroring consumes additional bandwidth on the VMs. Google quantifies it — if a mirrored instance experiences 1 Gbps of ingress traffic and 1 Gbps of egress traffic, the total traffic on the instances is 1 Gbps of ingress and 3 Gbps of egress (1 Gbps of normal egress traffic and 2 Gbps of mirrored egress traffic).
Correct approachBudget egress at three times the workload's own before enabling it, and filter aggressively from the start. #77 could say flow logs introduce no delay or performance penalty; this is the opposite trade, and it is paid on the machine you are trying to observe.
It is not, and this follows directly from where the mirror is taken. Mirrored traffic is encrypted only if the VM encrypts that traffic at the application layer. The platform encryption does not help, because the encryption and decryption happens in the hypervisors, and so from the perspective of the VM, this traffic is not encrypted. The mirror is taken on the VM's side of that boundary.
Correct approachTreat the collector fleet as a system holding full plaintext payloads of everything the mirrored workloads send and receive, and scope its access accordingly. If a workload handles regulated data and speaks a protocol without TLS internally, mirroring it duplicates that data onto machines whose threat model was probably never reviewed for it.
There is no such migration. On the workload level, the Network Security Integration Packet Mirroring takes precedence over VPC Packet Mirroring, and the scope of that precedence is not per-workload at all: the data plane disregards all VPC Packet Mirroring configurations for all workloads. It applies even when the Network Security Integration Packet Mirroring configuration is applied implicitly to the workload.
Correct approachPlan it as a single cutover with the full rule set in place before the first NSI rule exists. Google's own worked example is a team mirroring HTTP the new way and discovering that the ICMP traffic mirroring configured in the VPC Packet Mirroring policy is terminated, and ICMP traffic is no longer mirrored — with the understatement that this change in behavior can have significant implications on your network.
Architecture
Sources on one side, a load-balanced collector fleet on the other, and a policy that is not ordered the way anything else in this block is ordered.
Choosing sources, and the trap in choosing a subnet
Mirrored sources are Compute Engine VM instances that you can select by specifying subnets, network tags, or instance names. Those are the same three axes #72 established for firewall targeting, minus service accounts, and they compose permissively: you can specify one or more source types, and an instance matching any one of them is mirrored.
The subnet option carries a consequence worth stating plainly: if you specify a subnet, all existing and future instances in that subnet are mirrored. So a mirroring policy written once silently extends to every machine anyone deploys into that subnet afterwards, in plaintext, at triple egress. That is the right default for a security mandate and a poor one for a time-boxed investigation.
Coverage is per interface, not per machine: in cases where an instance has multiple network interfaces, the other interfaces aren't mirrored unless another policy has been configured to do so. That is the third post in a row where a multi-NIC VM needs enabling more than once. And the familiar container caveat returns — to mirror traffic passing between Pods on the same Google Kubernetes Engine (GKE) node, you must enable Intranode visibility for the cluster.
The collector, and the loop you must not build
A collector destination is an instance group that is behind an internal load balancer, reached through a forwarding rule configured specifically for mirroring. That specialisation has a sharp edge: any non-mirrored traffic that is sent to the load balancer is dropped. It is not a general-purpose load balancer that also takes mirrored traffic.
The placement rules are tight. All mirrored sources must be in the same project, VPC network, and Google Cloud region, and a collector destination must be in the same region as the mirrored sources — though it may sit in a peered VPC, which is what makes a centralised collector in a hub network possible. The cardinality is one-way: each mirroring policy can only reference a single collector destination, but however, a single collector destination can be referenced by multiple mirroring policies. So a central collector serving several networks means one policy per network, not one policy with several sources.
And the configuration error that looks reasonable: you cannot mirror and collect traffic on the same network interface of a VM instance because doing this would cause a mirroring loop. A collector that is itself inside a mirrored subnet would mirror the mirror. The subnet-wide source selection above makes this easy to do by accident, which is the argument for putting collectors in their own subnet from the start.
The ordering model is the opposite of every firewall rule in this block
This is the detail I found most striking. #70 established that for firewall rules priority is the only ordering and specificity never decides. Here, priority does not exist as a control: the priority of a packet mirroring policy is always 1000 and cannot be changed.
So overlapping policies resolve by specificity, using exactly the longest-prefix logic the firewall explicitly refuses. If policies have different but overlapping CIDR ranges and the same exact protocols, Google Cloud chooses the policy that uses the most specific CIDR range. Protocol breaks the next tie: if policies specify the same exact CIDR range with overlapping protocols, Google Cloud chooses a policy with the most specific protocol. And where there is no overlap at all there is no contest — if policies have the same exact CIDR range but distinct protocols, these policies don't overlap.
Then the honest admission at the end of that ladder. Identical policies are not supported, and if you build them anyway, Google Cloud can send traffic to any of the load balancers that have been configured with identical packet mirroring policies — worse, Google Cloud might choose the same policy or a different policy each time that matching traffic is re-evaluated against these policies. Documented non-determinism, in a security capture path.
An engineer who has internalised eight posts of "priority decides, specificity never does" will reason incorrectly about mirroring policies, and the failure is quiet: traffic goes to a collector, just not the one intended, and nothing errors. The practical rule from the documentation is to make policies distinct rather than relying on the tie-breaks — non-overlapping ranges, or unique protocols where ranges must overlap. Then the resolution order never has to be invoked.
What it cannot see, and what sees it
Two interactions with #77 matter. VPC Flow Logs doesn't log mirrored packets, so turning on mirroring does not inflate your flow log bill through the mirror traffic itself. But the collector is a normal destination for normal purposes: if a collector instance is on a subnet that has VPC Flow Logs enabled, traffic that is sent directly to the collector instance is logged, including traffic from mirrored instances. Flow logs on the collector subnet therefore produce an entry per mirrored flow, which is a straightforward way to double your logging costs without intending to.
The limits are small enough to design around rather than discover. Per policy: 5 subnets 5 tags 50 instances. And the filter budget is multiplicative, which is the part that bites — the maximum number of packet mirroring filters is 30, which is the number of IPv4 and IPv6 address ranges multiplied by the number of protocols. Google spells out the arithmetic: you cannot specify 30 ranges and 2 protocols, which would be 60 filters and greater than the maximum. Thirty ranges with one protocol is legal; fifteen ranges with two protocols is the equivalent budget.
On protocols it is the broadest instrument in the block: all layer 4 protocols are supported by Packet Mirroring, against #77's six and #76's two. Address families need help, though — by default, Packet Mirroring collects all IPv4 traffic of mirrored instances, and IPv6 requires an explicit filter, where you can mirror all IPv6 traffic by using a CIDR range filter of ::/0.
One last counting trap, the same shape as #77's: if two VM instances that are being mirrored send traffic to each other, Google Cloud collects two versions of the same packet. East-west traffic inside a mirrored subnet is captured twice, so volume estimates built from a single VM's traffic will understate the collector's load.
Why This Architecture Holds Up
Both of these change the decision, and the first one changes it before you start.
Google recommends a different product
The overview opens by pointing elsewhere. If you want to analyze your workloads' network traffic at scale and monitor network traffic using third-party virtual appliances, use Network Security Integration Packet Mirroring, and more directly: we recommend that you use Network Security Integration Packet Mirroring for new deployments.
What the replacement does differently is interesting in its own right, because it folds mirroring into the object this block spent five posts on. Packet Mirroring clones the network traffic based on the filtering criteria specified in the mirroring rules of the firewall policy — you can specify mirroring rules as part of your firewall policy. So mirroring stops being a separate policy type with its own ordering rules and becomes rules inside the global network firewall policy from #74, with the priority model that implies. It also gains a negative form, which the old product has no equivalent of: you can use the DO_NOT_MIRROR action in your mirroring rules to define negative filters.
| VPC Packet Mirroring | Network Security Integration Packet Mirroring | |
|---|---|---|
| Encapsulation | Packets are mirrored without encapsulation | Mirrored traffic is encapsulated using Geneve |
| Rule semantics | Mirroring rules are evaluated on a per-packet basis | Mirroring is session based |
| Negative filters | No | DO_NOT_MIRROR |
| Where the rules live | Its own packet mirroring policy | Global network firewall policy; regional network firewall policies don't support Packet Mirroring |
| Secure tags as a source | Network tags supported | Mirroring rules don't support source secure tags |
Two rows deserve a note. The Geneve encapsulation is not cosmetic — it adds a unique Virtual Private Cloud (VPC) network identifier to each packet redirected to the mirroring endpoint group for deep packet inspection, which is what makes a shared collector safe when several networks use overlapping RFC 1918 ranges. With the unencapsulated product, a central collector receiving the same private address from two peered networks cannot tell them apart. And the regional-policy row is the fourth capability this series has found missing from regional network firewall policies, after Layer 7 inspection in #74.
The performance picture is also worse than the old product's, and stated more bluntly. When network packets match any mirroring rule, Compute Engine processes both the original and mirrored packets at a slower rate, and then the recommendation that should decide platform choices: enabling mirroring is not recommended for 3rd generation machine families and later because the rate of the slower route cancels out their high-bandwidth networking benefits. So on modern machine types, mirroring and high-bandwidth networking are close to mutually exclusive.
You cannot run the two side by side. If a workload has an Network Security Integration Packet Mirroring configuration, the data plane disregards any existing VPC Packet Mirroring configuration — and the blast radius is total: the data plane disregards all VPC Packet Mirroring configurations for all workloads. It triggers even when the Network Security Integration Packet Mirroring configuration is applied implicitly to the workload, which means an NSI rule added at the policy level by someone else is enough. Write the complete replacement rule set first, verify it covers every protocol the old policies covered, and cut over once.
The cost, and why the cheap rate is misleading
The pricing shape is unusually simple. You are charged for the amount of data processed by Packet Mirroring, while you are not charged for Packet Mirroring forwarding rules and currently, there is no additional per-VM charge for using Packet Mirroring. The rate, in us-central1 and the regions grouped with it, is inbound data processed by load balancer for Packet Mirroring $0.008 / 1 gibibyte.
Compared with #76 and #77 that looks almost free. Those were vended logs — vended logs include VPC Flow Logs, Firewall Rules Logging, and Cloud NAT logs — billed twice at a combined $0.50 per GiB in the first tier. At $0.008, mirroring is roughly 62 times cheaper per gibibyte.
And that comparison is a trap, because the denominators are not comparable. Flow logs charge for a sampled, aggregated, metadata-trimmed summary; mirroring charges for every byte of every packet including payloads. A workload producing a few megabytes of flow logs a day produces its entire traffic volume in mirrored data. The per-unit rate is 62 times lower and the unit count is orders of magnitude higher, so the mirroring bill is the larger one in almost every real case — and that is before the collector VMs, which you pay for as ordinary compute, and before cross-zone transfer. For the NSI product the zone placement is called out explicitly: you are charged for cross-zonal traffic if the consumer and producer of your mirroring deployment are in different zones.
Which makes the practical cost model: filter hard, mirror narrowly, keep collectors zone-local to their sources, and treat subnet-wide mirroring as a deliberate, reviewed expense rather than a default. The $0.008 is the least of it.
Key Architecture Decisions
| Situation | Decision | Because |
|---|---|---|
| IDS, deep packet inspection, or PCI forensics | Packet Mirroring, not flow logs | Signatures need every packet of a flow; flow logs sample and estimate. |
| Starting a new mirroring deployment | Network Security Integration Packet Mirroring | Google recommends it for new deployments, and it brings Geneve and negative filters. |
| Moving from the old product to the new one | One cutover, complete rule set first | One NSI configuration makes the data plane disregard all VPC mirroring for all workloads. |
| Mirroring a regulated workload | Scope the collector fleet as a data store for that regulated data | The copy is plaintext unless the application itself encrypted it. |
| Sizing a mirrored instance | Budget three times the egress | Google's example: 1 Gbps in and out becomes 3 Gbps of egress. |
| Third-generation or newer machine families | Reconsider mirroring those workloads | The slower processing route cancels out their high-bandwidth networking. |
| Choosing sources for a time-boxed investigation | Instance names or tags, not a subnet | A subnet source silently includes every future instance. |
| Placing collector instances | Their own subnet, never a mirrored one | Mirroring and collecting on one interface creates a mirroring loop. |
| Flow logs on the collector subnet | Turn them off unless you need them | Traffic sent directly to a collector is logged, one entry per mirrored flow. |
| Two policies whose ranges overlap | Make them distinct instead of relying on tie-breaks | Priority is fixed at 1000, and identical policies resolve non-deterministically. |
| Needing more than 30 range-protocol combinations | Redesign the filter, or split policies | The filter cap is ranges multiplied by protocols. |
| A central collector for several networks | One policy per source network, and prefer the Geneve product | A policy references one collector, and only Geneve carries a network identifier for overlapping ranges. |
| IPv6 traffic | Add an explicit ::/0 filter |
The default collects IPv4 only. |
The one to look at today
For every packet mirroring policy you have, answer two questions. Which subnets are named as sources, and therefore which machines deployed since the policy was written are now being mirrored in plaintext at triple egress? And is there any Network Security Integration mirroring configuration anywhere in the project — because if one exists, every one of those policies is already being disregarded, and the capture you believe is running is not.
Closing Thought
Four posts have now circled one question: what does it cost to know what your network did? The answer this block has produced is a straight line. Firewall logs are cheap, capped and blind to most protocols. Flow logs are cheaper per unit of insight and twice sampled. Mirroring is complete, and it charges you in the workload's own bandwidth, in plaintext exposure, and in a collector fleet you operate. There is no instrument here that is complete, cheap and unobtrusive, and the useful skill is knowing which of the three you are giving up.
The specific thing I will carry forward is the encryption sentence, because it is a reminder that an abstraction's boundary is a real location. Google encrypts VM-to-VM traffic, and that is true and worth having. It happens in the hypervisor, and the mirror is taken on the other side of it, so the sentence "our VPC traffic is encrypted" and the sentence "our mirrored traffic is encrypted" are both about the same packets and only one is true. Eighteen posts into this block, that is the pattern in its purest form: the guarantee is real, the guarantee's boundary is documented, and almost nothing in the interface makes you look for it.
#79 leaves observability for structure: Shared VPC, host projects and service projects — the model whose host-and-service split has already turned up in three of the last four posts.
Official Google Cloud Reference
- Packet Mirroring overview
- Out-of-band integration overview
- VPC network pricing
- Google Cloud Observability pricing
- Configure Packet Mirroring
- Monitor Packet Mirroring
- Packet Mirroring partners
- VPC Flow Logs overview
- VPC quotas and limits
- Encryption in transit in Google Cloud
- Configure intranode visibility
Comments