AWS Daily Intelligence #27 - The appliance died and nobody told the client

Executive summary

Monday was Labor Day and AWS published nothing, so today's item is the strongest unwritten one from last week. On 3 September AWS added TCP Reset support to Gateway Load Balancer: GWLB now supports sending TCP Reset (RST) packets when a target becomes unhealthy, is deregistered, or when a flow's idle timeout expires.

The announcement frames this as faster recovery, and it is — AWS says traffic previously continued reaching unhealthy targets for several minutes because of TCP retry behaviour, and that endpoints can now recover in seconds. But the sentence worth sitting with is what the feature is not. It does not move your flow to a healthy appliance. The target group attribute that governs that, target_failover.on_unhealthy, still defaults to no_rebalance.

So the mechanism is not failover, it is notification. The dead flow stays dead. What changes is that the client is told, immediately, instead of retransmitting into silence until its own stack gives up. The client then opens a new connection, and that one is placed on a healthy target. Recovery happens because the client reconnects, not because AWS rescued the flow.

What changed

Aspect	Before	After
Unhealthy target, existing flow	Flow stays pinned to the target. Packets are dropped silently. The client retransmits until its own TCP timeout.	Optionally, GWLB sends a TCP RST. The client learns at once and can reconnect.
Deregistration	Target enters `draining` and waits out `deregistration_delay.timeout_seconds`, 300 seconds by default.	RST can be sent, so flows do not have to be waited out.
Idle timeout expiry	The flow state is discarded. The client discovers this only on its next packet, if at all.	RST is sent when the flow's idle timeout expires.
Default	Off. AWS states it is not enabled by default to ensure backward compatibility, and is set per target group through the console, CLI or API.
Scope and price	All new and existing Gateway Load Balancers, in every Region where GWLB is available, at no additional charge.

Architecture

Diagram: an appliance failure in a Gateway Load Balancer inspection VPC, before and after TCP Reset. Before, the appliance fails its health check, the existing flow stays pinned to it because target_failover.on_unhealthy defaults to no_rebalance, packets are dropped silently, and the client retransmits until its own TCP timeout, which AWS describes as several minutes. After, with TCP Reset enabled on the target group, the Gateway Load Balancer sends a reset packet, the client learns immediately, and the client opens a new connection which is placed on a healthy appliance. A note records that the flow itself is still not rebalanced and that recovery happens because the client reconnects. — What TCP Reset changes is who knows the flow is dead, and how quickly. The flow itself is still not moved — `no_rebalance` is still the default.

Gateway Load Balancer sits in the middle of a bump-in-the-wire inspection path, encapsulating the original packet in GENEVE on port 6081 and handing it to an appliance that decapsulates it, inspects it, and returns it. Because the appliance is stateful, GWLB pins each flow to one target, so every packet in a conversation reaches the same firewall. That is the entire point of it.

The pinning is 5-tuple by default: the target group attribute stickiness.enabled is false unless you set it, and AWS states that when it is false, 5-tuple is used. Configurable alternatives are source_ip_dest_ip and source_ip_dest_ip_proto.

That pinning is also what makes appliance failure awkward. The flow belongs to a specific target, and the two attributes that decide what happens when that target goes away both default to leaving the flow where it is:

target_failover.on_unhealthy — rebalance or no_rebalance, default no_rebalance.
target_failover.on_deregistration — the same values, the same default, and AWS is explicit that the two cannot be set independently: whatever you choose, both must match.

Leaving them at the default is a defensible choice, not an oversight. Rebalancing a live flow onto a different appliance hands it to a firewall that has never seen the beginning of the conversation, so a stateful inspection engine may simply drop it as out-of-state. Moving the flow can be worse than losing it. What was missing was any way to lose it promptly.

Business value

The value is bounded and easy to state: it converts a multi-minute stall into a fast, explicit failure. AWS's own framing is that traffic previously kept reaching unhealthy targets for several minutes because of TCP retries, and that endpoints can now recover in seconds.

For a payments or API tier behind an inspection VPC, the difference between a connection that hangs for minutes and one that is refused immediately is the difference between a stuck request queue and a retry. Client libraries almost always handle a reset well; almost none handle a silent black hole well.

Security considerations

Nothing here weakens the inspection guarantee, and that is worth being precise about. The reset is sent when a target is unhealthy, deregistered, or the flow has timed out — it does not cause traffic to bypass inspection. A new connection is placed on a healthy appliance and is inspected normally.

There is a small information-disclosure consideration in that an RST is a signal a client can observe, so a peer can distinguish "appliance failed" from "network black hole" more easily than before. For an internal inspection path this is not usually a meaningful concern, and the operational gain is large.

Cost considerations

No additional charge for the feature. The relevant cost is the one it makes visible rather than one it adds: a stalled flow occupies a connection slot, a worker thread, and often a retry budget for the whole of its timeout. Shortening that is a capacity saving that will not appear on the AWS bill.

Operational considerations

Enabling this changes what applications see, which is exactly why AWS shipped it off by default. A component that currently treats a stalled connection as "slow, keep waiting" will start receiving a hard refusal. That is the improvement, and it is still a behaviour change, so it belongs in a change window with the retry paths reviewed rather than switched on estate-wide on a Friday.

The deregistration case is the one to rehearse first, because it is the planned one. Replacing an appliance today means the target enters draining and existing flows wait out deregistration_delay.timeout_seconds — a 300-second default, which is five minutes of a maintenance window spent watching a timer. With reset enabled that window is bounded by how fast clients reconnect.

Tradeoffs

Option	Works well when	Breaks down when
TCP Reset on, `no_rebalance`	Clients reconnect cleanly. The common case, and the one AWS has now made viable.	A long-lived flow that cannot be re-established cheaply — a large transfer, a database session, a tunnel.
TCP Reset off, `no_rebalance`	Nothing. This is the old behaviour, and its only merit is that it is what you already have.	Any appliance failure: the stall lasts until each client's own TCP timeout.
`rebalance`	Appliances that are genuinely stateless, or that share state out of band.	Stateful inspection. The new appliance never saw the handshake and may drop the flow as out-of-state.

Implementation guidance

Check what your target groups do today before changing anything. The failover attributes are the ones that decide the shape of a failure, and most groups have never had them set:

bash

aws elbv2 describe-target-group-attributes \
  --target-group-arn <arn> \
  --query "Attributes[?starts_with(Key, 'target_failover') \
           || Key=='deregistration_delay.timeout_seconds' \
           || Key=='stickiness.enabled']"

Expect to see no_rebalance twice, 300, and false — the defaults. Enable TCP Reset per target group, verify against one appliance by failing its health check deliberately, and measure the client-side recovery rather than trusting the console to tell you it worked.

Best practices

Enable it, but per target group and behind a change window, because it changes what clients observe.
Leave target_failover at no_rebalance for stateful appliances. Reset plus reconnect is the safer path than moving a live flow to a firewall that never saw its handshake.
Remember the two failover attributes must carry the same value; you cannot rebalance on deregistration only.
Test the deregistration case first. It is the planned one, and it is where the 300-second default is costing you a real maintenance window.
Confirm your clients retry. A reset only helps a client that opens a new connection.

Who should adopt this

Anyone running a GWLB inspection path with stateful appliances — which is to say most people who run GWLB at all. The change is free, Region-wide, applies to existing load balancers, and addresses a failure mode that is otherwise invisible until an appliance dies during business hours.

The exception is a fleet whose flows are long-lived and expensive to re-establish. There, a reset ends the flow sooner but does not make re-establishing it cheaper, and the honest answer is that appliance failure will hurt either way.

Key takeaways

TCP Reset is notification, not failover. The flow is still not rebalanced; the client is simply told, so it can reconnect.
target_failover.on_unhealthy and .on_deregistration both default to no_rebalance, and must be set to the same value as each other.
Off by default for backward compatibility, per target group, no charge, every GWLB Region.
The planned case — deregistration with a 300-second default drain — is where the benefit is easiest to measure.
Monday produced no announcements. A day with no news is not a gap in coverage; the weekly roundup still carries the date range, and held items compete for the slot on their merits.