Business Challenge
The previous post ended with a queue that could not tell you which of its problems it had. This one is about the case where the answer is “one customer” — and about how little of that is decided at the moment anybody thinks they are deciding it.
A SaaS platform onboards a tenant several times larger than the rest. Nothing was mis-sized: the table has plenty of capacity, the queue has no throughput limit, the consumers autoscale. Latency for every other customer degrades anyway.
The isolation model was never chosen. It was inherited from two decisions made long before the large tenant existed.
The constraint people miss is not the table's throughput — it is the partition's. “Every partition in a DynamoDB table is designed to deliver a maximum capacity of 3,000 read units per second and 1,000 write units per second.”
If the partition key is the tenant identifier — the obvious, clean, universally taught choice — then that is a per-tenant ceiling. Provisioning the table for ten times the traffic does not raise it. A tenant large enough to want more than one partition's worth cannot have it, because their data is all under one key.
AWS states the design rule directly: “You should design your application for uniform activity across all partition keys.” A tenant-per-partition-key model is a bet that your customers are uniformly sized, which is a bet no SaaS business wins.
FixShard the key for tenants that outgrow a partition, before one of them does.
The ceiling is in units, not operations, and units are a function of size. AWS's own worked example makes the gap concrete: with a 20 KB item, “a single consistent read operation will consume 5 read units. This means you can concurrently drive 600 consistent read operations per second on that single item before reaching the partition limits.”
Six hundred, not three thousand. Nothing about the tenant changed — only the row got bigger. A schema that grows its items over time quietly lowers the ceiling for every tenant sitting behind that key.
FixRecompute the per-tenant ceiling whenever item size changes. It is size-dependent, not fixed.
A shared queue has no concept of a customer unless the producer supplies one. SQS fair queues
exist to mitigate exactly this — they “automatically mitigate the noisy-neighbor
impact in multi-tenant queues” — and they are enabled by one thing:
“message producers should add a tenant identifier by setting a
MessageGroupId on outgoing messages.”
No identifier, no tenants, no fairness. The mitigation is not something the consumer or the queue owner can switch on — it depends on a field the producing service decided whether to populate, possibly years ago, possibly in another team's codebase.
FixSet the tenant identifier at the producer. It costs nothing and cannot be retrofitted downstream.
Architecture
Pool, silo and bridge describe where tenants share infrastructure. The useful question is narrower: for each shared resource, what is the unit the platform uses to tell tenants apart, and what does it do with that knowledge?
In the table, the boundary is the key and the ceiling is fixed
The partition key is not merely how rows are found; it is the unit of throughput isolation. Two consequences follow that a capacity plan does not show.
The first is that a large tenant is capped independently of the table. The second is subtler and cuts the other way: several small tenants sharing a key share a ceiling, so a pooled design with a coarse key can produce contention between customers who are each individually tiny.
Adaptive capacity softens this — and note that it “applies to on-demand mode and provisioned capacity”, so it is not a provisioned-mode feature you can opt out of by choosing on-demand. It redistributes capacity toward busy partitions. It does not raise the per-partition maximum, so it helps with uneven load and not at all with a tenant whose steady state exceeds one partition.
In the queue, the boundary is a field somebody else populates
Fair queues detect a noisy neighbour “by monitoring message distribution among tenants during processing (the ‘in-flight’ state)”, and when one tenant holds “a disproportionately large number of in-flight messages compared to others”, SQS “prioritizes message delivery for other tenants.”
Two properties make this unusually easy to adopt. It “does not require any change in the consumer code”, has “no impact on API latency”, and carries “no throughput limitations”. And a tenant becomes noisy either by volume or by cost — AWS names both: sending more messages, or when “consumers take longer to process messages from that particular tenant.” The slow tenant is a noisy neighbour too, which is the case a rate limit at the producer would never catch.
This is the detail to carry out of the post. “MessageGroupId on standard queues with fair queues does not have the same behavior as MessageGroupId on FIFO queues. On standard queues, MessageGroupId is used only as a tenant identifier for fair queues and does not enforce message ordering.” Producer code that sets a group ID is requesting ordering on one queue type and tenant fairness on the other. Move a workload from FIFO to standard for throughput and the ordering guarantee disappears silently, because the field that expressed it is still there and still accepted.
Why This Architecture Holds Up
Fairness is prioritisation, not a quota
It would be easy to read fair queues as per-tenant throttling. AWS is explicit that it is not: “Amazon SQS does not limit the consumption rate per tenant. It allows consumers to receive messages from noisy neighbor tenants when there is consumer capacity and the queue has no other messages to return.”
That is the right design and worth understanding precisely. The noisy tenant is not punished and their work is not deferred indefinitely — they are simply no longer allowed to monopolise in-flight slots while others wait. When there is spare capacity, they use it. Fairness here means protecting the quiet tenants' dwell time, not capping the loud one.
The corollary matters for capacity planning: fair queues do not reduce total work, so they do not remove the need for the drain-rate decision from the previous post. They change who waits, not how much there is to do.
The guidance names the conditions: use fair queues when the queue is multi-tenant, high-throughput, and dwell time is part of your quality of service. The reasoning is honest about the limits: “At low throughput, one tenant's burst rarely creates a backlog that affects other tenants.” And it explains why scaling is not the answer at the top end: “over-scaling the consumer fleet to absorb every burst is impractical, and even with auto-scaling, the delay before new consumers come online can let backlogs form.” Fair queues fill the gap autoscaling cannot close — the minutes before capacity arrives.
The quiet-group metrics are the ones worth alarming on
The previous post argued that queue depth cannot distinguish its causes. In a multi-tenant queue the fair-queue metrics do exactly that, which is the strongest operational reason to adopt the pattern even before the mitigation matters.
ApproximateNumberOfMessagesVisibleInQuietGroups reports the backlog
“excluding messages from noisy message groups”, which AWS describes as helping
“identify the true processing backlog for typical message groups.” The
queue-level metric spikes when one tenant surges; the quiet-group metric stays flat, and the
difference between them is the entire diagnosis.
There is an equivalent for latency —
ApproximateAgeOfOldestMessageInQuietGroups, explicitly intended to
“set alarms for message processing timeouts that ignore artificially aged messages from
noisy neighbors” — and ApproximateNumberOfNoisyGroups
to count the offenders. An alarm on quiet-group age answers “are we failing our customers?”
The queue-level one only answers “is somebody busy?”
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Tenant as partition key | Only for uniformly small tenants | It sets a hard per-tenant ceiling of 3,000 read and 1,000 write units per second. |
| Large tenants | Shard the key, or silo them | No table-level capacity raises a partition maximum. The bridge model exists for exactly this tenant. |
| Item growth | Treated as a capacity change | A 20 KB item costs 5 read units, leaving 600 reads per second against that key rather than 3,000. |
| Adaptive capacity | Relied on for skew, not for scale | It redistributes toward busy partitions but does not raise the per-partition maximum. |
| Queue tenant identifier | Always set MessageGroupId |
Fairness is impossible without it, it costs nothing, and only the producer can supply it. |
| Queue type change | Re-read what the group ID means | On standard queues it is a tenant identifier only and does not enforce ordering. |
| Multi-tenant alarms | Quiet-group metrics | Queue-level depth and age both move when one tenant surges. Quiet-group metrics report the rest. |
| Expecting a quota | Do not | AWS does not limit consumption rate per tenant. Fair queues reprioritise; they do not throttle. |
Why this decision is almost always made by accident
Both boundaries are set in code that does not look like an isolation decision. A partition key is
chosen in a data-model discussion, on access-pattern grounds, when the platform has three customers
of similar size. A MessageGroupId is populated — or not — by
whoever wrote the producer, in a line that looks like metadata.
Neither review asks “what is the largest tenant this survives?” By the time that question is asked, the answer is already encoded in a key that a large amount of data is now stored under, and changing it is a migration rather than a setting. That is the honest argument for asking it early: the decision is cheap on the day it is made and expensive on every day after.
Closing Thought
Pool, silo and bridge is a useful vocabulary and a slightly misleading one, because it suggests the isolation model is a thing you select. In a pooled design the model is whatever the shared resources use to tell tenants apart, and those are ordinary-looking fields chosen for ordinary-looking reasons.
The partition key is the clearest case. It is taught as a data-modelling decision, and it is simultaneously a throughput ceiling, a contention boundary and a blast radius — 3,000 read units and 1,000 write units per second, fixed, regardless of what the table is provisioned for. Nobody writing that key is told they are setting a limit for a customer who has not signed yet.
The queue is the more encouraging case, because AWS has done the hard part. Fair queues need one field, no consumer changes, no throughput cost, and they bring metrics that finally separate “we are behind” from “one customer is busy.” The requirement is that somebody upstream populates the identifier — which is, once again, a decision that looks like nothing at the moment it is made.
Application patterns — event schema evolution: why a contract between two teams outlives the code on both sides of it, what actually breaks when a producer adds a field, and why the consumer that ignores unknown fields is doing the only thing that scales.
Comments