Business Challenge
The last seven posts have been about making a distributed system correct. This one is about a pattern that is usually correct and usually unnecessary, and about telling those two cases apart before the week of work rather than after it.
A team cannot answer a query efficiently. The write model is keyed for writes, the question is shaped differently, and the answer currently requires a scan. Somebody proposes CQRS: separate the read model from the write model, project changes into a store shaped for the question, query that.
That proposal is correct in principle and, most of the time, describes something the database already does. The gap between the two is worth seeing precisely.
Read the definition of a global secondary index against the definition of a read model and they do not diverge. It holds a selection of attributes, organised by a different key, and DynamoDB maintains it: “any global secondary indexes on that table are updated asynchronously, using an eventually consistent model. Applications never write directly to an index.”
Asynchronous projection, separate shape, no direct writes, eventual consistency. That is the pattern, implemented, with no projector to write, deploy, monitor or fix.
It even has the property teams usually cite as the reason for separating the two: “The provisioned throughput settings of a global secondary index are separate from those of its base table”, and a query against it consumes the index's capacity “not the base table”.
FixPrice the index first. Hand-build a read model only when a named requirement rules the index out.
Here is the sentence that changes how you provision, and it runs opposite to the direction anyone expects: “If you perform heavy write activity on the table, but a global secondary index on that table has insufficient write capacity, the write activity on the table will be throttled.”
And the general form: “For a table write to succeed, the provisioned throughput settings for the table and all of its global secondary indexes must have enough write capacity to accommodate the write. Otherwise, the write to the table is throttled.”
The read model is not merely allowed to lag. An under-provisioned one reaches back and stops the writes — which is precisely the coupling CQRS is sold as removing, present in the version you did not have to build.
FixProvision index write capacity at or above the base table's. AWS states this as the rule, not as advice.
An index is sparse by construction: “A global secondary index only tracks data items where its key attributes actually exist.” Write an item without the indexed attribute and “DynamoDB doesn't write any data for that item to the index.”
That is a genuinely useful feature — it is how you index only open orders — and it is a trap when nobody chose it. A count from the index and a count from the table legitimately disagree, and the difference is every item missing the attribute. AWS's own example makes it concrete: four items in the table, three in the index.
FixDecide whether sparseness is the design or an accident, and write down which.
Architecture
The decision is not “CQRS or not”. It is how much of the pattern you need, and there are three rungs, each with more capability and more to own.
What the built-in read model cannot do
The index is the right answer often enough that the limits are worth knowing exactly, because they are what legitimately forces the next rung.
It serves queries, not lookups. “The GetItem
and BatchGetItem operations can't be used on a global secondary
index.” Query and Scan only.
The projection is the contract. “Global secondary index queries cannot fetch attributes from the base table.” An attribute you did not project is not one query away — it is a second request against the table, per item.
Results are capped. “The maximum size of the results returned by a
Query operation is 1 MB.” A read model backing a report that
returns more than that is paginating, which is a different thing from answering.
Those three rule out full-text search, arbitrary aggregation, and joins across entities. Those are real requirements and they are the honest reasons to build a projector into OpenSearch, Athena or a relational store. “The pattern is cleaner” is not on that list.
Indexes are not uniformly expensive, and knowing which writes are costly changes the model. Changing an indexed key attribute “from A to B” costs two writes — “one to delete the previous item from the index and another write to put the new item into the index.” Meanwhile an update that touches “only attributes that are neither index key attributes nor projected into the index” consumes no index capacity at all, though the base table is still charged. So a status field used as an index key is the expensive one; a description field nobody indexed is free. Index the attributes that are queried, not the attributes that change.
Staleness is the design, not a defect
Every read model is behind the write model. AWS is specific about the normal case and honest about the tail: propagation happens “within a fraction of a second, under normal conditions”, but “in some unlikely failure scenarios, longer propagation delays might occur.”
And then the obligation, stated as an application requirement rather than a caveat: “your applications need to anticipate and handle situations where a query on a global secondary index returns results that are not up to date.”
This is the part of CQRS that cannot be engineered away by choosing a different read store. A hand-built projector has the same property and usually a longer lag. If the business rule is “this read must reflect that write”, no read model satisfies it — the answer is to read the write model, which for DynamoDB means a strongly consistent read on the table, something an index does not offer at all.
Why This Architecture Holds Up
The economics are decided by the projection, and they are visible
The three projection types are a straight cost dial. KEYS_ONLY
“results in the smallest possible secondary index”;
ALL “results in the largest possible”, with
INCLUDE between them. Storage is arithmetic you can do in advance: base
table key, plus index key, plus projected attributes, plus “100 bytes of
overhead per index item.”
Reads are cheaper than the table's, because they are eventually consistent by definition: an index read consumes “one half of a read capacity unit”, so a query “can retrieve up to 2 × 4 KB = 8 KB per read capacity unit.” AWS's worked example — eight items of 2,000 bytes, rounded to a 4 KB boundary and halved — comes to 2 read capacity units.
A hand-built read model has none of this legibility. Its cost is a Lambda, a destination store, a dead-letter queue and an on-call rotation, and none of those appear in a capacity calculator.
An under-provisioned index does not degrade gracefully into a slightly stale read model. It throttles writes to the base table — the whole table, not merely the indexed path. AWS flags this as affecting “all write operations, from indexing process to potentially disrupting your production workloads.” A read-side capacity mistake therefore presents as a write-side outage, which is the hardest kind of incident to attribute: the alarm fires on the table, the cause is on an index somebody added for a report.
Multi-attribute keys remove the commonest reason people gave up on indexes
The historical objection to indexes as read models was that one index answers one question, and real
queries have several dimensions — which drove teams to synthetic concatenated keys like
TOURNAMENT#WINTER2024#REGION#NA-EAST, and from there to giving up and
building a search index.
That objection has weakened. Multi-attribute keys allow “a partition key from up to four attributes and a sort key from up to four attributes, for a total of up to eight attributes per key schema”, without concatenation.
The constraint to design around is the query grammar: you “must specify all partition key attributes using equality conditions”, sort key attributes are matched left to right, and you “cannot skip attributes in the middle.” That is a hierarchy, not a set of independent filters — so the ordering of the sort key attributes decides which questions the read model can answer, and it is chosen once.
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Default read model | A global secondary index | Asynchronous, separately keyed, separately provisioned, and already maintained for you. |
| Index write capacity | At or above the base table's | An index short of write capacity throttles writes to the table, not just to itself. |
| Projection | The attributes the query returns | Queries cannot fetch from the base table, and ALL can double storage. |
| Indexed attributes | Queried, not frequently updated | Changing an indexed key attribute costs two index writes; changing an unprojected one costs none. |
| Sparseness | Deliberate, and documented | Items lacking the index key are absent entirely, so index and table counts legitimately differ. |
| Hand-built read model | Only for search, aggregation or joins | Those are what the 1 MB cap, the projection rule and Query-only access genuinely rule out. |
| Reads that must be current | Strongly consistent read on the table | No read model is current. An index offers no strongly consistent option at all. |
| Multi-dimensional queries | Multi-attribute keys, ordered deliberately | Up to eight attributes, but matched left to right with no gaps — the order fixes the questions. |
Where the pattern's reputation misleads
CQRS is discussed alongside event sourcing, and the two travel together often enough that adopting one is taken to imply the other. They are separable: an index is a read model with no event log behind it, and the write side remains a plain mutable table.
The second confusion is scale. CQRS is presented as a scaling pattern, and the scaling it buys is read-side: a shape that answers a question cheaply. It does not reduce write cost — it raises it, because “a table with many global secondary indexes incurs higher costs for write activity than tables with fewer indexes.” Every read model is paid for on the write path, whether the database maintains it or your projector does.
Closing Thought
The useful question is not whether to separate reads from writes. On DynamoDB that separation is available by adding an index, and most teams who describe themselves as considering CQRS are describing an index with extra steps.
The useful question is what the separation is being asked to do. If it is “answer this question without a scan”, the database does that already, with a projection you can price and a staleness measured in a fraction of a second. If it is full-text search, or aggregation, or a join, the index genuinely cannot — and those requirements are specific enough to name in a design document, which is a better justification than the pattern's reputation.
And whichever rung you land on, provision the read side properly, because the coupling runs the wrong way. A read model that cannot keep up does not quietly fall behind. On DynamoDB it throttles the writes, and the incident arrives labelled as a problem with the table.
Data platform — Kinesis, MSK or Firehose: three services that all move a stream, why the decision is usually made on operational ownership rather than throughput, and the retention and replay differences that decide whether a consumer can ever be fixed after the fact.
Official AWS Reference
- Using Global Secondary Indexes in DynamoDB — asynchronous maintenance, separate throughput, the write-throttling rule, projections and write amplification
- Change data capture for DynamoDB Streams — further reading for the hand-built read model path
- Troubleshooting throttling in Amazon DynamoDB — further reading on diagnosing the write-side symptom
Comments