Kinesis Data Streams now has queues: consumers receive records instead of leasing shards. The cost saving is real and the announcement leads with it, but the thing to understand first is what the queue stops promising.
“A queue also does not necessarily deliver records in the order that producers wrote them to the stream.”
Executive summary
“Amazon Kinesis Data Streams now supports queues, a simpler and more cost-effective way to process a stream”, and “Queues in Kinesis Data Streams are powered by Amazon SQS”. They are “available for data streams in On-demand capacity mode”, and the documentation lists 31 Regions at launch — a broad commercial footprint with no GovCloud or China Regions in the table.
The mechanism: “A queue delivers records from an On-demand Kinesis data stream to a queue consumer and tracks the delivery state of each record”, so “your queue consumer does not manage shards, leases, or checkpoints”. And the consequence, stated plainly: “Queue consumer instances do not lease shards: each one receives the next visible record, so you add instances to process more records in parallel without resharding.”
That is the whole trade in two sentences. A shard preserves order and caps your parallelism at the shard count. A queue removes the cap and gives up the order.
What changed
Until now, consuming a Kinesis stream meant one consumer instance per shard. The documentation states the guarantee that bought you: “Each shard is read by one stream consumer instance at a time, which preserves the order of the records in that shard.” It also states the ceiling, since a shard supports “up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second”, and your consumer count is bounded by your shard count.
The machinery for managing that was the Kinesis Client Library, and the documentation is candid about what it costs you operationally: “Kinesis Client Library uses Amazon DynamoDB tables to store metadata related to data consumption. It creates three tables per application that is processing data.” Three DynamoDB tables, lease coordination, checkpoint state, and a rebalance every time an instance comes or goes.
A queue deletes that layer. The announcement quantifies it: “Kinesis Data Streams consumers can reduce the cost of processing data by up to 43% by using queues instead of the Kinesis Client Library”, and names where it comes from — “You no longer run a DynamoDB table or custom metrics, and you use compute more efficiently.”
It is also additive rather than a migration: “A queue works alongside your existing stream consumers and requires no changes to your producers.” One stream can feed a stateful stream consumer and a stateless queue consumer at the same time.
Architecture
Shard lease against queue delivery
| Stream consumer | Queue consumer | |
|---|---|---|
| Unit of work | A shard, leased | A record, delivered |
| Ordering | “preserves the order of the records in that shard” | None promised |
| Parallelism ceiling | The shard count | “without resharding” |
| Delivery | You checkpoint position | At least once; unacknowledged records redelivered |
| Sequence number | Returned with each record | Not returned |
| State you run | Leases and checkpoints, three DynamoDB tables with KCL | None — the queue tracks delivery state |
The sequence number row is the one that tells you this is a different primitive rather than a cheaper client. “A queue does not return the sequence number of the records it delivers, because a sequence number identifies a record's position within its shard, and a queue consumer is decoupled from shards.” Any code that logs, correlates or deduplicates on sequence number has nothing to use.
The decision rule, which AWS states twice
The documentation defines the two cases rather than leaving it to judgement. Stateful: “a consumer processes records in sequence to keep an accurate state, because processing one record depends on the records before it” — live analytics, personalization, anomaly detection. Stateless: “a consumer processes each record independently, because processing one record does not depend on any other record” — the announcement's examples are “log processing, order fulfillment, and notifications”.
Worth pausing on “order fulfillment” as a stateless example, because it is the one most likely to be misread. Fulfilling an order independently of other orders is stateless. Processing state transitions of a single order — created, paid, shipped — is not, and a queue can deliver those three records in any sequence. The test is not whether the domain sounds event-driven; it is whether two records about the same entity can be handled in either order without changing the outcome.
At-least-once, with an unusual honesty
“A queue delivers each record at least once: it redelivers any record that is not acknowledged before its visibility timeout expires, and in rare cases it can deliver a record again after you acknowledge it.”
That second clause is rarer to see written down than the first. A consumer that treats acknowledgement as a guarantee of exactly-once has a window it cannot close, which is why the instruction is explicit: “Design your queue consumer to be idempotent, so that processing the same record more than once is safe.”
The API shape pushes you the right way: “ReceiveQueueRecords, which returns up to 10 records at a time” and “AcknowledgeQueueRecords, up to 10 per call”. Acknowledgement is per record, not per batch position, so a partial failure in a batch of ten does not force you to replay the nine that worked.
The dead-letter state is not a holding area
Dead-letter handling is built in — “a record that is received too many times without an acknowledgment becomes a dead-letter record that you handle through a separate set of operations”, with “MaxReceiveCount, from 1 to 1,000” deciding how many attempts that takes.
And then the sentence that should change how you plan: “A queue can deliver a record only while the stream still retains it, so a record that is never acknowledged, including a dead-letter record, expires when the stream's retention period ends for that record.”
This is unlike a standard SQS dead-letter queue, where a failed message sits until you deal with it. Here the dead-letter state is a flag on a record that still lives in the stream, and the stream's retention is the clock. If retention is 24 hours, your window to investigate and reprocess a poison record is under 24 hours from when it was written — not from when it failed. A dead-letter record discovered on Monday from a Sunday write may already be gone.
The limit that will surprise a fan-out estate
“A queue counts as one registered consumer and shares the stream's registered consumer limit with enhanced fan-out consumers.”
If you already run several enhanced fan-out consumers on a stream, each queue you add competes for the same budget. That is a sensible accounting, and it is not visible from the queue documentation's own framing of queues as a lighter-weight option — lighter operationally, identical in consumer-slot cost.
Business value
The clearest value is for consumers that were never stateful in the first place. A large share of Kinesis consumers do per-record work — enrich and forward, write to a store, call a model, send a notification — and have been paying for shard leasing, KCL coordination and three DynamoDB tables per application to get it. The announcement's the up-to-43% figure is the headline, and the operational saving of deleting lease and checkpoint logic is the part that compounds.
The scaling change is the deeper one. Parallelism no longer requires resharding, so a consumer that falls behind is an autoscaling problem rather than a stream reconfiguration. Anyone who has resharded a production stream to clear a backlog will recognise what that removes.
And because a queue “requires no changes to your producers” and works alongside existing stream consumers, the migration unit is one consumer rather than the whole pipeline.
Security considerations
Nothing about the data path changes. A queue reads a stream you already own, so stream encryption, KMS key permissions and producer identity are unchanged. Server-side encryption still requires that “producer and consumer applications must have permission to access the AWS KMS key”, and a queue consumer is a consumer for that purpose.
The audit trail loses a correlator. Without sequence numbers, the identifier you used to trace a specific record through processing is gone, and what replaces it has to come from your own payload. If an investigation has ever depended on identifying a record by its sequence number within a shard, decide what the equivalent is before moving a consumer.
Idempotency is now a security property, not only a correctness one. A record that can be delivered again after acknowledgement means any side effect that grants, charges or notifies can happen twice. For a notification that is noise; for an entitlement change it is not.
Cost considerations
The announcement's figure is comparative and bounded: “Kinesis Data Streams consumers can reduce the cost of processing data by up to 43% by using queues instead of the Kinesis Client Library”. Treat it as the ceiling of a saving whose size depends on how much of your current bill is KCL overhead rather than stream throughput.
Three components actually move. The DynamoDB tables KCL creates — “three tables per application” — go away. The compute no longer has to be sized to the shard count, so an over-provisioned consumer fleet can shrink to its real work. And the queue itself is a priced feature, so check the pricing page rather than assuming the delta is pure saving.
One cost shape worth noticing: because scaling no longer requires resharding, the lever that used to cost you shard-hours now costs you consumer instances. That is usually cheaper and always more reversible.
Operational considerations
On-demand only. “Queues are available for data streams in On-demand capacity mode.” A provisioned stream has to change capacity mode before it can have a queue, which is a stream-level decision with its own cost consequences.
Set MaxReceiveCount against your retention, not against your patience.
A high retry count on a short-retention stream can consume the entire window before anyone looks at the
dead-letter record.
Alarm on the dead-letter path immediately. “Each queue publishes its own CloudWatch metrics, so you can scale consumer instances on records waiting to be received and alarm on records that repeatedly fail” — and the dead-letter state is both the signal that something is poisoned and a countdown against stream retention.
Check your registered consumer headroom before adding queues, since each one spends a slot shared with enhanced fan-out.
Region availability is narrower than usual. The documented list is 31 Regions, with no GovCloud or China entries, so a multi-partition design cannot assume parity yet.
Tradeoffs
| Choice | What you gain | What it costs |
|---|---|---|
| Queue consumer | Scale past the shard count; no leases or checkpoints; built-in dead-letter | No ordering, no sequence numbers, mandatory idempotency |
| Stream consumer | Per-shard ordering and sequence numbers | Parallelism capped by shards; leases, checkpoints and KCL's tables |
| Both on one stream | Stateful and stateless consumers from the same data, no producer change | Each queue spends a registered consumer slot |
High MaxReceiveCount |
More chances for a transient failure to clear itself | Retries can exhaust the retention window before a human sees the record |
Implementation guidance
Apply the entity test, not the domain test. Ask whether two records about the same entity can be processed in either order with the same result. If not, it is stateful regardless of how independent the workload sounds.
Make the consumer idempotent on a key from the payload, because the sequence number is not available and acknowledgement is not a guarantee.
Start with one consumer, not the pipeline. A queue runs alongside existing stream consumers, so the lowest-risk first move is the single most clearly stateless consumer you have.
Compute your real dead-letter window as stream retention minus the age of the record when it first failed, and set alarms to fire well inside it.
Confirm the stream is On-demand before designing anything around queues.
Best practices
Treat a queue consumer and a stream consumer as different programs rather than two configurations of one. The absence of ordering and sequence numbers changes what the code can assume, and a consumer ported without that review is a consumer whose correctness now rests on luck.
Keep the dead-letter handling code written before you need it. Dead-letter records are handled “through a separate set of operations” and have a deadline, which is a bad combination to discover during an incident.
Record somewhere durable why each consumer is on a queue or on a shard. The next person's temptation will be to move everything to the cheaper option.
Who should adopt this and when
Adopt now for genuinely per-record consumers of on-demand streams. Log shipping, notification fan-out, per-record enrichment and inference are the cases where you were paying for ordering you never used.
Adopt now if resharding to add consumer capacity is a familiar operation. That whole class of work disappears.
Do not adopt for anything stateful, and read “stateful” strictly: any consumer where two records about one entity must be handled in sequence.
Wait if you are in GovCloud or China, or on a provisioned stream, where the feature is either unavailable or requires a capacity-mode change first.
Key takeaways
A queue removes the shard as the unit of consumption, and with it the two things the shard provided: “does not necessarily deliver records in the order that producers wrote them”, and no sequence number, “because a sequence number identifies a record's position within its shard, and a queue consumer is decoupled from shards”. In exchange you “add instances to process more records in parallel without resharding”.
The dead-letter state is a flag, not a destination: “a record that is never acknowledged, including a dead-letter record, expires when the stream's retention period ends for that record.” Your investigation window is the stream's retention, counted from the write.
And a queue is not free of the consumer budget — it “counts as one registered consumer and shares the stream's registered consumer limit with enhanced fan-out consumers”.
Official AWS references
- Amazon Kinesis Data Streams announces queues to simplify and reduce costs of real-time processing — the announcement: queues as a simpler and more cost-effective way to process a stream, powered by Amazon SQS, the up-to-43% processing cost reduction against the Kinesis Client Library, the stateless and stateful use-case lists, and the On-demand capacity mode requirement
- Consume records from a stream with queues — that a queue works alongside existing stream consumers with no producer changes, that it counts as one registered consumer sharing the limit with enhanced fan-out, the
ReceiveQueueRecordsandAcknowledgeQueueRecordsten-record shapes, the at-least-once guarantee including redelivery after acknowledgement, the instruction to design for idempotency, the absence of ordering, built-in dead-letter handling, the retention ceiling on dead-letter records, and the 31-Region launch list - Amazon Kinesis Data Streams terminology and concepts — the queue and queue consumer definitions, stateful versus stateless processing, that each shard is read by one stream consumer instance at a time preserving order, that queue consumer instances do not lease shards, that a queue returns no sequence number, the
MaxReceiveCountrange of 1 to 1,000 and the dead-letter state, the per-shard throughput limits, and that the Kinesis Client Library creates three DynamoDB tables per application - Quotas and limits for queues — further reading on the per-queue and per-stream limits; no claims in this post are drawn from it
- Monitor a queue — further reading on the CloudWatch metrics each queue publishes for scaling and alarming; no claims in this post are drawn from it
Comments