Homeβ€Ί Blogβ€Ί AWS Architecture Series #69 β€” Small records cost more, and the managed path goes down to zero…
AWS Architecture AWS Architecture Series

AWS Architecture Series #69 β€” Small records cost more, and the managed path goes down to zero seconds

Streaming versus batch gets argued as a latency question, and the cost side is assumed to follow from volume. It does not. The managed path bills per record rounded up to 5 KB, and its buffering range is set by the destination rather than by the service, so the decision turns on record size and on whether anything but the destination needs to read the stream.

Verified against current vendor documentation on 1 October 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

#60 covered which streaming service to pick. This is the cost lens on the same ground: what the choice actually bills for, and what the buffering range does and does not decide.

Corrected 1 October 2026

The first version of this post treated 60 seconds as Firehose's minimum buffer interval across the board, on the strength of a single sentence in the quota page, and built the architecture argument on that floor. That is wrong. The buffer interval is destination-specific, and is documented as 0 to 900 seconds for Amazon S3, Apache Iceberg tables, Redshift, OpenSearch and several HTTP destinations. The sections below are the corrected argument.

The usual framing is a trade between latency and cost, with volume as the driver. Two documented facts change the shape of it.

1Firehose bills per record, rounded up to 5 KB

The quota page states the pricing model directly, which pricing pages rarely do: “Firehose ingestion pricing is based on the number of data records you send to the service, times the size of each record rounded up to the nearest 5KB (5120 bytes).”

So a 100-byte event is billed as 5,120 bytes — over fifty times its own size. Shrinking that event to 50 bytes saves nothing. Batching two events into one record halves the bill.

AWS gives the worked example: “if the total incoming data volume is 5MiB, sending 5MiB of data over 5,000 records costs more compared to sending the same amount of data using 1,000 records.” Same bytes, different bill.

Fix

Optimise record count, not payload size. The rounding unit makes small events the expensive case.

2The buffering range belongs to the destination, not to Firehose

Firehose buffers before delivering, and the quota page carries one flat sentence about it: “The buffer interval hints range from 60 seconds to 900 seconds.” Read on its own, that looks like a service-wide floor. It is not one, and treating it as one is how this decision gets made wrongly.

The API reference is the authority on the range, and it gives BufferingHints.IntervalInSeconds a valid range of “Minimum value of 0. Maximum value of 900”, with a default of 300. The same 0-to-900 range is documented for OpenSearch, for HTTP endpoints, and for Snowflake — whose default is zero. The FAQ states it per destination in the same terms: “You can configure the values for S3 buffer size (1 MB to 128 MB) or buffer interval (0 to 900 seconds).”

So sub-minute delivery is a supported configuration on the managed path — for Amazon S3, Apache Iceberg tables, Redshift, OpenSearch and several HTTP destinations — rather than a reason to leave it. It is not free: below a 60-second interval Firehose uses multipart upload to S3 to get the latency, and that arrives as additional S3 PUT requests on the bill. A cost, not a wall.

Fix

Look up the buffering range for the destination you are actually writing to. Reach for Data Streams because the application needs to consume the stream itself, not because of a floor.

3The 60%-cheaper streaming mode is a reservation

On-demand Advantage reads like a free upgrade: “Data ingest, data retrieval, and extended retention usage across all on-demand streams are at least 60% lower than in On-demand Standard”, and “there's no longer a fixed charge for each stream.”

Then the condition: “Enabling On-demand Advantage commits the account to at least 25MiB/s of data ingest and 25MiB/s of data retrieval across all on-demand streams.” Under-use it and “you'll be charged the difference as a shortfall.”

And it is account-level with a lock-in period: “a minimum period of 24 hours before you can disable the mode.” A serverless-shaped product with a floor, a shortfall charge and a cooling-off period is a reserved-capacity deal.

Fix

Measure sustained ingest and retrieval against 25 MiB/s each before enabling. Below it you pay for air.

Architecture

The curve has two axes and only one of them has a step in it. Cost steps hard, at 5,120 bytes per record. Latency does not: on the managed path it runs down to zero, and what takes you off that path is needing to consume the records yourself rather than have them delivered somewhere.

Diagram: the latency and cost curve between streaming and batch on AWS, drawn from documented quotas rather than from data volume. At the low-latency end sits Kinesis Data Streams with a consumer application you operate yourself, reaching sub-second delivery. It offers three modes. On-demand Standard requires no capacity planning and accommodates up to double the peak write throughput observed in the previous thirty days, with write throttling possible if traffic more than doubles the previous peak within fifteen minutes. On-demand Advantage is an account-level mode where ingest, retrieval and extended retention are at least sixty percent lower than Standard and the per-stream fixed charge disappears, but enabling it commits the account to at least twenty-five mebibytes per second of ingest and twenty-five of retrieval, bills any shortfall at the discounted rate, and cannot be disabled for twenty-four hours; it also raises enhanced fan-out consumers from twenty to fifty per stream with no price premium. Provisioned mode requires specifying shards, sized by the formula ceiling of the maximum of incoming write bandwidth in kibibytes divided by 1024 and outgoing read bandwidth divided by 2048. A panel notes that on-demand splits shards evenly when traffic rises but explicitly does not detect or isolate hot hash keys, so AWS recommends provisioned mode for granular shard splits, and that a single partition key exceeding one megabyte per second or one thousand records per second will still throttle. At the managed end sits Amazon Data Firehose, whose buffer interval is documented as zero to nine hundred seconds with a default of three hundred, so the floor belongs to the destination rather than to the service and sub-minute delivery stays on the managed path at the cost of multipart uploads and extra S3 PUT requests. Firehose ingestion is billed by the number of records times each record's size rounded up to the nearest five kilobytes, which is 5,120 bytes, so a hundred-byte event is charged for more than fifty times its own size, and AWS's own example shows five mebibytes sent as five thousand records costing more than the same five mebibytes sent as one thousand records. A closing panel records the design consequence: optimise record count rather than payload size, check the buffering range for the destination you are writing to rather than assuming a service-wide floor, and measure sustained throughput against the twenty-five mebibyte commitment before enabling the discount.
Two axes, both documented: record count sets the bill, buffer interval sets the floor.

Three Kinesis modes, and they are not three prices

“A mode determines how the capacity of a data stream is managed and how you're charged for the usage of your data stream.” That sentence is the whole point — the mode is a billing model, not a performance tier.

On-demand Standard “accommodates up to double the peak write throughput observed in the previous 30 days”, which is a useful and under-appreciated guarantee: your headroom is a function of your own history. The matching warning is that “write throttling can occur if your traffic increases to more than double the previous peak within a 15-minute duration.”

Provisioned has an actual formula, which is rare enough to quote: number_of_shards = ceiling(max(incoming_write_bandwidth_in_KiB/1024, outgoing_read_bandwidth_in_KiB/2048)). Note the asymmetry — reads get twice the divisor, because a shard offers 2 MB/s out against 1 MB/s in.

And on-demand explicitly cannot fix a hot key

This inverts the usual assumption that the serverless mode is strictly better. “In the on-demand mode, Kinesis Data Streams splits the shards evenly when it detects an increase in traffic. However, it does not detect and isolate hash keys that are driving a higher portion of incoming traffic to a particular shard. If you are using highly uneven partition keys you may continue to receive write exceptions. For such use cases, we recommend that you use the provisioned capacity mode that supports granular shard splits.” So skew is the one case where you are told to leave the managed mode β€” and it is the case Service-Managed Partition Keys addressed by removing the key entirely, at the cost of ordering.

Where the rounding unit actually bites

The 5,120-byte unit is benign for anything log-shaped and brutal for anything event-shaped. A 2 KB application log line rounds to one unit and wastes 60% of it. A 100-byte IoT reading rounds to the same unit and wastes 98%.

Which means the aggregation you do before Firehose is worth more than any compression you do inside it, and PutRecordBatch is the lever — it “can take up to 500 records per call or 4 MiB per call, whichever is smaller”. That is batching for the API quota; packing several events into one record is what changes the bill.

There is a second, quieter version of the same trap on the far side of the pipeline: “If the increased quota is much higher than the running traffic, it causes small delivery batches to destinations. This is inefficient and can result in higher costs at the destination services.” Raising a Firehose quota speculatively makes your S3 object count go up and your query performance go down — the small-files problem #31 was about, arriving from a quota setting.

Why This Architecture Holds Up

Chaining the two removes a quota entirely

Direct PUT into Firehose is capped: “500,000 records/second, 2,000 requests/second, and 5 MiB/second” in the three largest Regions, and “100,000 records/second, 1,000 requests/second, and 1 MiB/second” elsewhere — a fivefold regional difference worth knowing before you design for one Region and deploy in another.

Put a data stream in front and the cap disappears: “When Kinesis Data Streams is configured as the data source, this quota doesn't apply, and Amazon Data Firehose scales up and down with no limit.”

That is the argument for the two-service pipeline rather than the one-service one, and it is a capacity argument rather than a feature argument. You are buying the stream's elasticity, and paying for the stream.

The buffer the managed path gives you for free, and its limit

“Each Firehose stream stores data records for up to 24 hours in case the delivery destination is unavailable and if the source is DirectPut.” Twenty-four hours of destination outage absorbed with no work from you. But read the condition: if the source is DirectPut. Chain a data stream in front β€” the thing the previous section recommends β€” and that guarantee moves: “If the source is Kinesis Data Streams (KDS) and the destination is unavailable, then the data will be retained based on your KDS configuration.” So the fix for the throughput quota changes who owns the retention window, and a 24-hour default on the stream is the number to check.

Consumer count is a pricing dimension, not just a limit

Enhanced Fan-Out “supports adding up to 20 consumer applications” on the standard modes. On-demand Advantage raises that to “up to 50 consumers per stream” and removes the surcharge: “Enhanced fan-out data retrievals also do not have a price premium compared to the standard data retrievals in this mode.”

So for a stream fanning out to many independent consumers, the commitment can pay for itself twice over — on the 60% discount and on the removed EFO premium. That is the case where Advantage is genuinely the right answer rather than a trap, and AWS says so: it suits accounts that “need many fan-out consumers, or operate with hundreds of data streams.”

Mode switching is reversible, within limits

“For each data stream in your AWS account, you can switch between the on-demand and provisioned modes twice within 24 hours… Switching between modes doesn't cause any disruptions to your applications.”

Twice a day, no disruption, is unusually generous — most of the decisions in this series have been one-way. The exception is the account-level one: On-demand Advantage needs 24 hours before it can be turned off, and “you must first remove any warm throughput” from the streams that carry it.

Key Architecture Decisions

Decision Choice Reasoning
Cost optimisation target Record count, not payload size Billing rounds each record up to 5,120 bytes.
Sub-minute delivery to a supported destination Firehose, buffer interval under 60s The documented range is 0 to 900 seconds; the cost is multipart upload and more S3 PUTs.
Up to 15 minutes Firehose, buffer tuned The managed path, and the whole documented interval range.
The application must consume the stream itself Data Streams with your own consumer Firehose delivers to destinations; it is not a consumer API, at any interval.
On-demand Advantage Only above 25 MiB/s sustained, both ways Shortfall is billed, and the mode locks for 24 hours.
Many fan-out consumers On-demand Advantage 50 consumers instead of 20, and no EFO price premium.
Skewed partition keys Provisioned, not on-demand On-demand splits shards but does not isolate hot hash keys.
Above the Direct PUT quota Chain a data stream in front The Direct PUT cap stops applying entirely.
Destination outage tolerance Check which source you use 24 hours is the DirectPut guarantee; with KDS it follows your stream retention.

The two numbers that settle it

Average record size, and required latency in seconds. Those two place you on the curve before any discussion of volume or service preference.

Under 5 KB per record, your bill is a function of how many records you emit, so aggregation upstream is the highest-leverage change available — and it is free. The latency number then places you within the managed path's range rather than outside it, because for the common destinations that range starts at zero; going under 60 seconds costs extra PUT requests, not a different architecture. What genuinely moves you to Data Streams is a consumer, not a number.

Closing Thought

Cost-lens posts usually end with a break-even. This one ends with a rounding unit, because that is where the money actually goes: 5,120 bytes per record whether the record is 5,000 bytes or fifty.

It is a good example of a cost structure that does not reward the intuitive optimisation. Compressing payloads, trimming fields, shortening keys — all of that is effort spent below the rounding unit and therefore invisible on the invoice. Emitting half as many records is the change that matters, and it is usually a batching change in the producer rather than anything in the pipeline.

The latency side turned out to be the opposite kind of lesson, and a useful one about reading documentation. One sentence in the quota page gives a 60-to-900-second range with no destination attached to it, and it is an easy sentence to promote into a property of the service. The API reference and the FAQ both document the range starting at zero. So the boundary between a service you configure and a service you operate is not a number at all — it is whether anything besides the destination needs to see the records.

Next in this series

Data & Analytics — the next unwritten roadmap entry. Items 21 to 33 are already published as posts #30 through #60, so the series continues past this block rather than revisiting it.

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent