Business Challenge
The last five posts built an event pipeline: idempotent consumers, events that exist at all, polite retries, a drain rate, tenant isolation. This one is about the thing that outlives all of it — the shape of the message, which is a contract between two teams and usually the only artefact that survives both of their rewrites.
A producer team adds an optional field. It is the most defensive change available: nothing removed, nothing renamed, the new field ignorable. A consumer breaks in production that afternoon.
Three things are going on, and the third is the one nobody expects.
“Compatible” is not a property of a change. It is a property of a change relative to a deployment order, and the registry makes you choose which order you are promising.
BACKWARD — AWS's recommended default — “allows consumers to read both the current and the previous schema version”, for when you “delete fields or add optional fields.” It assumes consumers upgrade first.
FORWARD “allows consumers to read both the current and the subsequent schema versions”, for when you “add fields or delete optional fields.” It assumes producers upgrade first.
Note that adding a required field is forward-compatible and not backward-compatible. Teams that assume “adding is safe, removing is dangerous” have it backwards half the time, because which one is safe depends entirely on who deploys first.
FixChoose the mode from your deployment order, not from intuition about additions and removals.
BACKWARD checks the new version against the previous one. FULL covers “the previous or next version of the schema, but not earlier or later versions.”
So a sequence of individually-compatible changes can leave version 6 unreadable by a consumer
still on version 2 — and every step passed its check. The _ALL
variants exist precisely for this: BACKWARD_ALL “allows consumers to read both the
current and all previous schema versions.”
If any consumer can lag more than one version — and a consumer reading a 24-hour stream replay always can — the pairwise modes are not the guarantee you wanted.
FixUse an _ALL mode if consumers can be more than one version behind.
Here is the rule that inverts most people's instinct. In JSON under BACKWARD, adding an optional
property registers “as long as the original schema version does not allow any additional
properties by setting the additionalProperties field to
false.”
And it is rejected when the original schema “sets the
additionalProperties field to true, namely allowing any additional
property… as they cannot read data with phone number property in a different type, for
example string instead of number.”
The logic is sound once you see it. If the old schema accepted anything, then old messages may already contain a field with that name and some other type. Declaring it now, with a type, is not a compatible narrowing — it is a claim about data that already exists and may contradict it.
FixClose the schema. additionalProperties: false is what keeps it evolvable.
Architecture
The registry is a small piece of machinery with one job: refuse a version that would break the promise you selected. Understanding it is mostly understanding what the promise covers.
_ALL variants are the only ones that promise anything about a consumer more than one version behind.The checkpoint decides what “previous” means
This is the mechanism most teams never look at, and it quietly determines what the check is actually comparing against. “A schema version that is marked as a checkpoint is used to determine the compatibility of registering new versions of a schema”, and on creation “the default checkpoint will be the first version.”
Then the detail worth writing on a wall: “In the console, editing the schema definition or compatibility mode will change the checkpoint to the latest version by default.”
So someone opening the console to adjust a setting can move the reference point of every future compatibility check. Nothing failed, nothing warned, and subsequent versions are now validated against a newer baseline than the one your oldest consumer is running. It is a governance control whose baseline can be relocated by a UI visit.
Optional is a type, not an annotation
The modes talk constantly about optional and required fields, and in Avro the definition is mechanical:
“An optional field is one in which the Type includes null.
Required fields do not have null as the Type.”
That is worth stating because “optional” in most schema conversations means “the consumer can cope without it”, which is a judgement. Here it is a union with null, present or absent, and the registry will accept or reject your change on that basis regardless of what the team meant.
Support is “AVRO (v1.11.4)”, JSON Schema “Draft-04, Draft-06, and Draft-07”, and “Protocol Buffers (Protobuf) versions proto2 and proto3 without support for extensions or groups.” Three things follow. JSON Schema 2019-09 and 2020-12 are not on that list. A proto2 file using extensions is outside what the registry validates. And in proto3 all fields are optional, which changes which compatibility rules can ever fire — a proto3 schema cannot fail a check for adding a required field, because it cannot have one.
Why This Architecture Holds Up
The registry encodes the argument teams would otherwise have
The genuine value is not validation. It is that “these modes form the contract between applications producing and consuming data” — a single declared answer to “who has to deploy first?”, stored next to the schema rather than in the memory of whoever set the pipeline up.
AWS states the outcome as making “data consumers resilient to compatible upstream changes”, and the word doing the work is compatible. The registry does not make consumers resilient; it defines which changes they have already agreed to tolerate, and refuses the rest at registration rather than at 3am.
Both are legitimate and both are frequently chosen by accident. NONE means “any new version added will be accepted without undergoing a compatibility check” — AWS suggests it for development, and it is what a schema ends up on when somebody needs a change to go through today. DISABLED “prevents versioning for a particular schema. No new versions can be added”, which looks like maximum safety and is really a decision to do evolution somewhere else, usually by creating OrderPlacedV2 and leaving the old topic to rot. Neither is wrong. Both should be deliberate, because both convert a checked contract into an unchecked convention.
gRPC inverts the intuition again, in a useful way
The Protobuf rules include a case that makes the direction concrete: “adding new RPC service or RPC method is a backward compatible change”, while “removing an RPC service or RPC method is a forward-compatible change.”
Adding is backward-compatible; removing is forward-compatible. The same pairing as fields, and the same trap — a team on FORWARD mode will find that adding a method is the change that gets rejected, which is the opposite of what “forward” sounds like it should mean.
It is free, which removes the last excuse
“The Schema registry is serverless and free to use.” The quotas are generous enough to disappear for most estates: 100 registries per Region, 10,000 schema versions per Region, 170KB per schema payload.
The one to notice is 10,000 versions per Region across the account, not per schema. A pipeline that registers a version on every deployment — which is exactly what auto-registration does — consumes that pool at the rate of your release cadence, not your schema-change cadence.
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Default mode | BACKWARD | AWS's recommendation, and it matches the common order: upgrade consumers, then producers. |
| Lagging consumers | BACKWARD_ALL | Pairwise modes say nothing about a consumer two versions behind. Replays always are. |
| Producer-first rollouts | FORWARD | Adding a required field is forward-compatible, not backward-compatible. |
| JSON schemas | additionalProperties: false |
An open schema cannot later declare a field, because old data may hold that name at another type. |
| Optionality | Union with null, explicitly | In Avro, optional means the type includes null. Intent is not what the check reads. |
| Console edits | Treat as a governance change | Editing the definition or mode in the console moves the checkpoint to the latest version. |
| NONE | Development only, with an expiry | It accepts every version unchecked. It is the mode a deadline selects. |
| Auto-registration | Watch the version count | 10,000 versions is per Region per account, and consumption tracks release cadence. |
The rule that makes all of this cheaper
Everything above is about what the registry will accept. The consumer still has to be written, and there is one behaviour that survives every mode: ignore fields you do not recognise. A consumer that deserialises strictly and fails on an unknown property has made every additive change — the safest category there is — into a breaking one for itself, regardless of what the contract permits.
That is the asymmetry worth internalising. The producer's obligation is to make compatible changes. The consumer's obligation is to tolerate the ones the contract already allowed. Only the second is free, and only the second scales to consumers you do not know about.
Closing Thought
Schema evolution gets discussed as a serialisation topic, which makes it sound like a library choice. It is a coordination problem wearing a format's clothing: two teams, two deployment schedules, and a message that has to be readable across the gap between them.
The compatibility mode is where that gets written down, and it is genuinely a design decision rather
than a setting. BACKWARD says consumers go first. FORWARD says producers go first. The
_ALL variants say we do not know how far behind anyone is. Picking one
without knowing which you meant is how a defensive change breaks production on a Tuesday.
And the JSON rule is worth carrying beyond the registry, because it contradicts a habit. An open schema feels generous and future-proof. What it actually does is forfeit your ability to say anything definite later, because you have already promised that the field could be anything. Constraints are not the opposite of flexibility here. They are the thing that preserves it.
Application patterns — the saga: why distributed transactions are not coming back, what a compensating action can and cannot undo, and why the hardest part of a saga is the step whose compensation is an apology rather than a rollback.
Comments