Business Challenge
An order platform runs a nightly Glue job. It reads the previous day’s events from S3, joins them against a dimension table, writes Parquet, and a BI tool picks it up at 06:00. Ten DPUs, about six minutes a run, thirty runs a month. Nobody has ever looked at the line item.
Then the operations team asks for the same numbers “in real time”, because a fulfilment decision that currently waits until tomorrow costs money today. The request is reasonable and the answer sounds simple: replace the nightly job with a stream.
The estimate that follows is where this goes wrong. The team prices the change the way it prices everything else — per gigabyte. We move roughly 400 GB a day, streaming is a bit more expensive per gigabyte than batch, so call it double. That model is not slightly off. It is the wrong model, and it is wrong in both directions at once: it overestimates what the stream costs to carry the data, and it completely misses the two terms that actually decide the bill.
The question is not “streaming or batch”. It is how fresh, and read by how many. Those are two different axes, they are priced by two different mechanisms, and only one of them has anything to do with how much data there is.
Architecture
Freshness on AWS is not a dial you turn smoothly from twenty-four hours down to one second. It is three regimes with different billing mechanics, and moving between them changes which term dominates.
Regime one: batch, measured in hours
AWS Glue bills DPU-hours. A single Data Processing Unit (DPU) provides 4 vCPU and 16 GB of
memory, and you are billed in increments of 1 second, rounded up to the nearest second, with
a 1-minute minimum duration per run. AWS’s own worked example on the pricing page:
AWS will bill you 6 DPU * 0.25 hour * $0.44, or $0.66.
The property that matters here is the one nobody writes down: between runs, this costs nothing at all. Cost is runs × DPU-hours, and the idle term is zero.
Regime two: micro-batch, measured in minutes
Amazon Data Firehose buffers and delivers. For Amazon S3 the buffering size range is 1–128 MB with a default of 5, and the buffering interval range is 0–900 seconds with a default of 300. There is no shard to provision and no hourly charge for having the stream exist — you pay per GB ingested, and a Firehose stream carrying nothing costs nothing.
This regime is where most “real time” requests actually land, and it is by some distance the cheapest way to buy a large improvement in freshness. Going from a nightly job to a five-minute buffer is a change of two orders of magnitude in staleness, and the billing mechanism does not change shape at all.
Regime three: real time, measured in milliseconds
Kinesis Data Streams provisions capacity. One shard costs $0.015 per
hour, and each shard can support up to 1 MB/sec or 1,000 records/sec write throughput
or up to 2 MB/sec or 2,000 records/sec read throughput. That hourly charge accrues whether or
not a byte is written.
On the read side, each shard can support up to five read transactions per second, and each read transaction can provide up to 10,000 records with an upper quota of 10 MB per transaction. Five transactions per second is the whole shard’s budget, shared by every consumer polling it — which is the mechanism behind the number in the next section.
The step between regime two and regime three is not gradual. AWS documents the exact point: Firehose uses multi-part upload for S3 destination when you configure a buffer time interval less than 60 seconds to offer lower latencies. Due to multi-part upload for S3 destination, you will see some increase in S3 PUT API costs. Sixty seconds is a cliff edge in the pricing model, and it is written into the buffering hints documentation rather than into the pricing page.
Why This Architecture Holds Up
Write the two cost functions down and the whole argument falls out.
Batch
cost = runs × DPU-hours × rate
Freshness is runs. It is a multiplier on the whole expression, so halving staleness doubles the bill. Volume appears only inside DPU-hours, and for a job whose runtime is dominated by startup rather than by data, it barely appears at all.
Streaming
cost = shards × hours × (1 + fan-out) × rate
Freshness does not appear. Neither does volume, except as the thing that decides how many shards you need. What multiplies is hours, which is fixed at 720 a month, and consumers.
The batch curve is steep because the minimum is not the data
That ten-DPU job costs $0.44 a run. Run it nightly and the month is $13.20. Run it hourly and the month is $316.80. Run it every five minutes and it is over $3,800. The data has not changed. What changed is that a job with a 1-minute minimum duration per run, whose real work takes six minutes, is being asked to pay its startup cost 8,640 times instead of 30.
The streaming line is flat because you already bought the whole hour
A single provisioned shard costs $10.80 over a thirty-day month, and it costs that whether the consumer reads every 200 milliseconds or once a day. Freshness in regime three is free at the margin because it was included in the purchase.
Which puts the crossing point somewhere most teams would not guess. The transport floor of a single shard is worth about 24.5 runs of a job billed at $0.44 — so a nightly job, at thirty runs, has already spent more than the stream it was chosen instead of.
Read that comparison carefully, because it is a scale comparison and not a substitution. The shard moves bytes; the Glue job transforms them. Replacing the job with a stream does not remove the compute — something still has to process what comes off the shard. The point is narrower and more useful: the transport layer stops being the deciding term almost immediately. Teams routinely reject streaming to avoid a $10.80 line item while paying $316.80 for the run frequency that was supposed to be the cheap option.
The term that actually makes streaming expensive
Fan-out. AWS publishes the number and it is not subtle: message propagation delay is an average of around 200 ms if you have one consumer reading from the stream, and this average goes up to around 1000 ms if you have five consumers. Adding consumers to a shared-throughput stream degrades latency five-fold, and costs nothing, because they are all dividing the same five read transactions per second.
Buying it back is enhanced fan-out. It is typically an average of 70 ms whether you have one consumer or five consumers, and an enhanced fan-out pipe provides up to 2 MB/sec of data per shard, independently of any other pipes or of the total number of consumers. It is also where the money goes: enhanced fan-out costs an extra $0.015 per shard, plus a charge of $0.013 per GB on retrievals.
Five enhanced fan-out consumers on a single shard cost $54.00 a month — five times what the shard itself costs. The fan-out is more expensive than the stream.
| Configuration | Propagation delay | Monthly, one shard | What you are paying for |
|---|---|---|---|
| One shared-throughput consumer | ~200 ms | $10.80 | The shard hour, nothing else |
| Five shared-throughput consumers | ~1000 ms | $10.80 | The same shard, five ways — latency is the price |
| Five enhanced fan-out consumers | ~70 ms | $64.80 plus retrievals | Dedicated 2 MB/sec pipes, one per consumer |
Key Architecture Decisions
Five teams each registering an enhanced fan-out consumer against the same stream is five consumer-shard-hour charges for data that is identical in all five cases. The propagation delay each of them needs is usually not the same, and often only one of them needs milliseconds at all.
Design rule
One enhanced fan-out consumer for the path that genuinely needs 70 ms. Everything else reads a Firehose delivery of the same stream from S3. You pay for the low-latency pipe once, and the four consumers who were going to poll a dashboard get a five-minute buffer that costs no shard-hours at all.
The buffering interval range starts at zero, and when you configure buffering interval as zero seconds, Firehose will not buffer data and will deliver data within a few seconds. It is one number in a console field, so it gets set low by whoever is testing and then never revisited. Below 60 seconds the delivery mechanism changes to multi-part upload and the S3 PUT API costs rise with it.
Design rule
Sixty seconds is the floor unless somebody can name the decision that the extra fifty-nine seconds changes. If they can name it, that path belongs in regime three on a real stream, not in a Firehose buffer tuned down until it approximates one.
Provisioned mode is the cheaper rate and the wrong default for anything bursty. The shard
hour accrues at 3 a.m. on a Sunday for a stream that only carries traffic during business
hours. On-demand removes the shard decision entirely — by default, new data streams
created with the on-demand capacity mode have 4 MB/s of write and 8 MB/s of read
throughput, and in US East (N. Virginia) they scale up to 10 GB/s of write and 20 GB/s
read throughput — but it carries its own hourly charge:
Per-stream charges = 30 * 24 * 0.040 (rate) = $28.80.
Design rule
On-demand while the traffic shape is unknown or spiky; provisioned once it is steady and measured. Both have a floor, and neither floor is avoidable by sending less data. The only way to stop paying is to not have the stream.
Extended data retention costs an extra $0.020 per shard hour, against a shard that costs $0.015 per hour. Turning on seven-day retention more than doubles the stream’s floor, and it is usually enabled to cover a replay scenario that S3 already covers — because the same records were being delivered to S3 by Firehose the whole time.
Design rule
Retention on the stream buys you replay for consumers that read the stream. If the replay story is “reprocess last Tuesday”, that is an S3 and Athena job, and it is cheaper by an order of magnitude. Extend retention for consumer outages measured in hours, not for reprocessing measured in days.
A job billed at one DPU-hour of $0.44 per run costs $316.80 a month when it runs hourly. The instinct when a report is stale is to schedule the job more often, and each doubling of frequency doubles the bill for processing the same data. With a 1-minute minimum duration per run, a job whose useful work is seconds long pays for a minute regardless.
Design rule
Past roughly hourly, stop increasing the schedule and change the regime. A Firehose stream into S3 with a five-minute buffer delivers fresher data than a fifteen-minute Glue schedule, with no per-run minimum to pay 2,880 times a month.
Next in this series: Amazon Data Firehose uses at-least-once semantics for data delivery, and delivery retries by Amazon Data Firehose might introduce duplicates if the original data-delivery request eventually goes through. Every design in this post therefore hands the consumer a record it may have seen before. Post #35 takes up idempotency — the property that makes at-least-once delivery safe to build on, and the one distributed systems cannot do without.
Closing Thought
The reason the per-gigabyte estimate fails is that it answers a question about volume when the bill is a question about time. Batch charges for work and stops; streaming charges for readiness and does not. A stream is a standing offer to deliver a record within 200 milliseconds, and you pay for the standing offer whether or not anyone takes it up.
Which reframes the design conversation usefully. “How fresh do you need this?” is the wrong opening question, because the honest answer is always “fresher”. The question that produces an architecture is: what decision changes if this arrives in one second rather than five minutes, and who is making it? If there is a decision and a person, you are in regime three and the enhanced fan-out charge is the price of the answer. If there is not, five minutes costs almost nothing — and one second costs what five minutes would have, every hour, forever.
Comments