Executive summary
Amazon Kinesis Data Streams has launched Service-Managed Partition Keys, aimed at “eliminating hot partition keys and reducing time to production for streaming workloads”. Instead of the producer choosing a key, “the service automatically distributes records based on available warm capacity”.
It covers “On-Demand Standard and On-Demand Advantage streams”, is “available today in all AWS commercial regions”, arrives “at no additional cost”, and is adopted by upgrading the SDK or the Kinesis Producer Library rather than by changing stream configuration.
The qualifying clause is the whole story, and it is easy to read past: this is for workloads “where record ordering is not required.” That is not a footnote. In Kinesis the partition key has always done two jobs at once, and you cannot hand over one without handing over the other.
What changed
A sentence in the developer guide has been true since the service launched: “When an application puts data into a stream, it must specify a partition key.” That requirement is what has just become optional.
What the key did mechanically: “An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards using the hash key ranges of the shards.” Deterministic, and indifferent to how much traffic any one value carries.
That indifference is the hot-shard problem. A key chosen for meaning —
customer_id, device_id,
tenant — inherits whatever skew the business has. One large customer
is one hash value, and one hash value is one shard.
Nothing about the shard limits changed. A shard still supports “up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second”. What changed is who decides which shard a record lands on.
Architecture
The second job is the one that does not appear in the announcement's headline, and it is stated in the developer guide's definition of a sequence number: “Each data record has a sequence number that is unique per partition-key within its shard… Sequence numbers for the same partition key generally increase over time.”
That is the ordering guarantee, and it is a consequence of the key, not a separate feature. All records sharing a key hash to one shard, and a shard is “a uniquely identified sequence of data records”. Order is preserved because the records were co-located, and they were co-located because they shared a key.
So “eliminate hot partition keys” and “preserve per-entity ordering” are not two features you can have separately. They are the same mechanism read two ways, which is exactly why the announcement scopes itself to workloads “where record ordering is not required.”
Business value
The work this removes is real and unglamorous. Hot-shard mitigation usually means salting the key with a random suffix, then reassembling meaning downstream — a change that touches the producer, the consumer, and anything that reasoned about the key in between.
“Reducing time to production” is the honest framing. For a team whose records are genuinely independent — clickstream, metrics, logs, telemetry fan-in — picking a partition key has always been a chore with no good answer, because the requirement being encoded was “spread evenly” and the key is a poor way to say that.
“Based on available warm capacity” is the other half. A hash function cannot know which shards are busy; a service can. That is a better input than any key the producer could have chosen, for the subset of workloads that do not need the key's other property.
Security considerations
Nothing in the security model changes: the same stream, the same KMS-backed server-side encryption, the
same IAM on PutRecord and PutRecords.
One second-order effect is worth naming. Partition keys are frequently business identifiers, and “partition keys are Unicode strings, with a maximum length limit of 256 characters” — a field that has carried customer IDs, account numbers and email addresses into stream metadata for years. Removing the requirement to supply one removes a place that identifier was being written.
That is a small, genuine privacy improvement for anyone who has had to think about what their key values reveal. It is not a reason to adopt the feature, but it is a reason not to reintroduce a key out of habit.
Cost considerations
“At no additional cost”, and restricted to on-demand streams, where “you are charged only for the actual throughput that you use.”
The indirect saving is the one that matters. Under provisioned mode a hot shard is paid for twice: once in the throttling, and once in the shards added to relieve it, since “you are charged for the number of shards at an hourly rate” whether or not they carry traffic. Even skew is cheaper capacity.
There is a detail in the shard limit worth carrying into any sizing estimate: “1 MB per second (including partition keys)”. The key counts against the byte budget. And the two write ceilings meet at roughly 1 KB per record — below that size a shard exhausts its 1,000-records-per-second limit before its megabyte, which is the case most telemetry workloads are actually in.
Operational considerations
Adoption is a client-library change, not a stream setting. “Simply upgrade to the latest AWS SDK or Kinesis Producer Library (KPL) version.” That is convenient and it is also how this reaches production without a review: a dependency bump plus a producer that stops passing a key is a small diff with a semantic consequence.
The failure it introduces is silent. A hot shard announces itself — throttling errors, a visible skew in per-shard metrics. Records arriving out of order relative to each other does not announce itself. It surfaces later, as a consumer that computed a wrong running total or applied a stale update over a fresh one.
Check the consumer, not the producer. The question is not whether the producer needs
ordering; it is whether anything downstream assumes it. Aggregations by entity, state machines,
last-write-wins updates and anything keyed off sequence number ordering all
quietly depend on the property being given up.
It is per-record, so the decision can be mixed. Nothing requires a stream to be
all-or-nothing; the key is supplied per PutRecord. Ordered entities can keep
explicit keys on the same stream as independent records that do not.
Tradeoffs
Even distribution against per-key ordering. The single real trade, and it is not tunable — the mechanism that provides one provides the other.
Service knowledge against determinism. “Available warm capacity” beats a hash function at balancing load and is no longer reproducible: you can no longer compute which shard a record went to, which matters for anyone who has built debugging around that.
On-demand only. Restricted to On-Demand Standard and On-Demand Advantage, so the teams on provisioned mode — often precisely those managing shard counts by hand because of skew — are not the ones who get this.
Implementation guidance
Write down what each consumer assumes about order before changing any producer. This is the entire decision, and it lives downstream of the change being made.
Adopt per record type, not per stream. Telemetry and audit events can stop specifying a key while entity updates on the same stream keep theirs.
If you are salting keys today, this replaces that. A random suffix is a worse version of what the service now does with knowledge of warm capacity — and it leaves reassembly logic downstream that can now be deleted.
Keep watching per-shard metrics through the change. It is the evidence that the distribution improved, and it is the only place the change is visible at all.
Best practices
Treat “record ordering is not required” as a claim to verify, not a description to accept. It is a statement about every consumer of the stream, including ones added later.
Do not reintroduce a partition key for meaning. If a record needs an entity identifier, that belongs in the data blob, where it is not also a routing decision.
Revisit sizing at roughly 1 KB per record. That is where a shard's record-count ceiling and its byte ceiling cross, and it decides which one you are actually limited by.
Who should adopt this
Available today everywhere commercial, free, and behind a library upgrade — so the barrier is verification rather than availability.
Adopt it now if your stream carries genuinely independent records and you have ever salted a key or added shards to relieve skew. Do not adopt it if any consumer aggregates or sequences by entity, if you are on provisioned mode where it is not offered, or if you cannot yet name every consumer of the stream — because the question this feature asks is about all of them.
Key takeaways
- Service-Managed Partition Keys removes the requirement that a producer specify a partition key.
- The service distributes records based on available warm capacity rather than by MD5 hash.
- On-Demand Standard and On-Demand Advantage only; all commercial Regions; no additional cost.
- Adopted by upgrading the AWS SDK or the KPL — a client change, not a stream setting.
- The partition key also provides ordering: sequence numbers are unique per partition key within a shard.
- So the feature is scoped to workloads "where record ordering is not required" — the trade is not optional.
- Shard limits are unchanged: 1,000 records/sec and 1 MB/sec writes, including partition keys.
- Those two ceilings meet near 1 KB per record; smaller records hit the count limit first.
- The new failure mode is silent — out-of-order processing surfaces as wrong results, not as errors.
Comments