The one you cannot re-read

Business Challenge

The application-patterns block is finished. This one opens the data platform section with the decision that gets made first and revisited least: which service carries the stream.

It is normally decided on throughput. Every one of the three will carry what a typical workload produces, so the comparison resolves to a shrug and whichever is most familiar.

The question that actually separates them arrives three weeks later, when somebody notices the consumer has been writing the wrong field since Tuesday. Whether that is a redeploy or an incident depends on a property nobody compared.

1Only two of the three keep the data

Kinesis Data Streams is a buffer you can re-read. Retention runs from a minimum of 24 hours to a maximum of “8760 hours (365 days)”. Fix the consumer, rewind, reprocess.

MSK is Apache Kafka, so retention is a topic setting you own, along with everything else.

Firehose is neither. It buffers “to a certain size or for a certain period of time before delivering it to destinations” — which is batching, not retention. Once delivered, there is nothing to go back to.

Fix

Pick on replay, not throughput. It is the property that differs most and gets compared least.

2The transform runs before the only copy

Firehose can transform records in flight, and that is most of its appeal — no consumer to write. But the transform runs on the way to the destination, and the destination is the only copy.

A wrong transform is therefore not a bug you redeploy past. It is a backfill, from whatever source the producer still has, if it has one. Firehose does offer to “back up source data to another Amazon S3 bucket” when transformation is enabled — which is the feature that makes the mistake survivable, and it is off unless somebody turned it on.

Fix

If Firehose transforms, enable source backup. It is the difference between a redeploy and a data-loss conversation.

3The default output is not what you would expect

This one costs an afternoon the first time. For S3, Firehose “concatenates multiple incoming records based on the buffering configuration”, and then: “By default, Firehose concatenates data without any delimiters.”

A thousand JSON records arrive as one object with no line breaks between them — which Athena, Glue and most line-oriented tooling cannot read. Newline delimiters are available as a setting, and the setting is off by default.

Fix

Enable the newline delimiter when you create the stream, not when the first query fails.

Architecture

The three are not three sizes of the same thing. They are a buffer, a broker, and a pipe, and the operational surface differs as much as the retention does.

Diagram: comparing Kinesis Data Streams, Amazon MSK and Amazon Data Firehose on the axis that separates them, which is retention and replay rather than throughput. Kinesis Data Streams is a buffer that can be re-read, with a retention period running from a minimum of 24 hours to a maximum of 8,760 hours or 365 days, so a consumer found to be wrong can be fixed and the stream reprocessed from an earlier point. Its provisioned shards each support one megabyte per second or a thousand records per second of writes and two megabytes per second or two thousand records per second of reads, while on-demand streams begin at four megabytes per second of write throughput and scale far higher in three Regions than elsewhere. Amazon MSK is open-source Apache Kafka, where AWS provides the control plane for creating, updating and deleting clusters while the customer uses Kafka data-plane operations and owns topic configuration including retention, with a minimum of one broker per Availability Zone. Amazon Data Firehose is a pipe rather than a buffer: it buffers incoming data only to a size or a time interval before delivering it, so there is no retention and nothing to re-read, and a transform applied in flight runs on the way to what becomes the only copy of the data. A panel records two Firehose defaults that surprise people: for Amazon S3 delivery it concatenates records without any delimiters unless newline delimiters are enabled, and it uses at-least-once semantics so retries may introduce duplicates for every destination type except Amazon S3, Apache Iceberg Tables and Snowflake. — A buffer, a broker and a pipe. The throughput numbers rarely decide it; the retention column always does.

Kinesis: the buffer, with a shard as the unit of everything

In provisioned mode the shard is the unit of capacity, and the numbers are worth holding: each shard supports “up to 1 MB/sec or 1,000 records/sec write throughput or up to 2 MB/sec or 2,000 records/sec read throughput.” Reads and writes are not symmetrical, and the read side is shared across consumers unless you register them.

On-demand removes the arithmetic and starts small: “4 MB/s of write and 8 MB/s of read throughput”, scaling to 10 GB/s write and 20 GB/s read in N. Virginia, Oregon and Ireland — and to 200 MB/s and 400 MB/s everywhere else. That is a fifty-fold regional difference in the same capacity mode, and it is the kind of thing that turns a working design into a support ticket when a workload moves Region.

Fan-out has a ceiling too: 20 registered consumers per stream, or 50 with On-demand Advantage. And switching capacity modes is rationed — “twice within 24 hours” — so it is not a dial to turn during an incident.

The read path has its own back-pressure, and it is easy to trip

A shard allows “five read transactions per second” and “a maximum total data read rate of 2 MB per second”. Ask for too much at once and you are throttled by design: “If a call to GetRecords returns 10 MB, subsequent calls made within the next 5 seconds throw an exception.” A replay — catching up on a day of retained data — is exactly the workload that hits this, so the recovery path is rate-limited precisely when you are in a hurry. Worth knowing before the incident rather than during it.

MSK: Kafka, with the control plane taken care of

The division is stated plainly: MSK “provides the control-plane operations, such as those for creating, updating, and deleting clusters” and “lets you use Apache Kafka data-plane operations, such as those for producing and consuming data.” Crucially it “runs open-source versions of Apache Kafka”, so existing tooling works unchanged.

That last property is the real reason to choose it, and it is an organisational one rather than a technical one: a team that already runs Kafka keeps its consumers, its libraries, its mental model and its runbooks. A team that does not is adopting Kafka's operational surface in exchange for configurability it may never use.

The surface is genuinely smaller than self-managed — KRaft controllers are “included at no additional cost… and require no additional setup or management”, and MSK Serverless goes further, managing brokers so “you only provision your Kafka server resources at a cluster level.” But topics, partitions, retention and consumer groups remain yours, and that is the point of choosing it.

Why This Architecture Holds Up

Firehose is the right answer more often than its limitations suggest

Everything above reads as an argument against Firehose, and it should not. For the case it was built for — get this data into S3 or OpenSearch, reliably, with nothing to operate — there is no consumer to write, no shards to size and no cluster to patch. That is a real reduction in surface, and most log and telemetry pipelines want exactly it.

The mistake is reaching for it when the requirement is processing rather than delivery. Delivery is idempotent in its useful sense: getting the same bytes to the same bucket twice is recoverable. Processing is not, and a pipe with no retention gives you nowhere to stand when the processing was wrong.

The duplicate semantics have a specific exception list

Firehose “uses at-least-once semantics for data delivery”, and retries “might introduce duplicates if the original data-delivery request eventually goes through.” The exceptions are named: this applies to every destination “except for Amazon S3 destinations, Apache Iceberg Tables, and Snowflake destinations.” So a Firehose stream into OpenSearch or an HTTP endpoint needs an idempotent consumer on the far side — the same requirement post #52 established for SQS, arriving here from a service that looks like it has no consumer at all.

The decision is about who operates it, not what it can carry

Line the three up by operational surface and the choice usually makes itself.

Firehose has none: no capacity to provision, no consumer to run. Kinesis has some: a shard count or a capacity mode, a consumer application, and a set of read-path limits to design around. MSK has the most: topics, partitions, replication factors, consumer groups, broker sizing — the full Kafka model, minus the control plane.

Each step up buys capability that the step below cannot provide, and the capability is almost always about time. Firehose gives you now. Kinesis gives you the last day to the last year. MSK gives you whatever you configure, plus the ecosystem that assumes you have it.

Key Architecture Decisions

Decision	Choice	Reasoning
Deciding axis	Replay, not throughput	All three carry a typical workload. Only two let you reprocess after finding a bug.
Delivery only, no processing	Firehose	No capacity to size, no consumer to operate. The surface really is close to zero.
Processing, or replay needed	Kinesis Data Streams	Retention from 24 hours to 365 days is the thing that makes a wrong consumer survivable.
Existing Kafka estate	MSK	It runs open-source Kafka, so tooling and consumers carry over unchanged.
Firehose with a transform	Enable source backup to S3	The transform runs on the way to the only copy. Backup is what makes an error recoverable.
Firehose to S3	Turn on newline delimiters	The default concatenates records with no delimiter, which most query tooling cannot read.
Kinesis capacity mode	Decided deliberately	You may switch only twice in 24 hours, and on-demand ceilings differ 50-fold by Region.
Non-S3 Firehose destinations	Idempotent consumer	At-least-once applies everywhere except S3, Iceberg Tables and Snowflake.

Two numbers that shape a design more than they look

Record size differs by an order of magnitude. A Kinesis record payload may be “up to 10 MiB”; a Firehose record “can be as large as 1,000 KB.” A payload that fits comfortably in one will be rejected by the other, which is the sort of thing discovered during a migration rather than during a comparison.

A shard iterator expires in five minutes. A consumer that fetches an iterator, does slow work, and comes back to use it finds it invalid. That is a design constraint on the consumer loop, not a tuning parameter, and it is the most common first bug against Kinesis.

Closing Thought

Streaming comparisons are usually written as capability matrices, and a matrix flatters the service with the most rows. MSK wins that format and is the wrong answer for most teams reading it, because the rows it wins are configuration options somebody has to own.

The useful question is narrower. Not “which can carry this?” — all of them can — but “what happens on the day we find out the consumer was wrong?” With Kinesis you rewind. With MSK you rewind, having configured how far back that goes. With Firehose there is nothing to rewind to, and the answer is a backfill from whatever the producer still holds.

That is not an argument against Firehose. It is an argument for noticing which of the three you are choosing, because the decision is normally made on throughput, and throughput is the axis on which they differ least.

Next in this series

Data platform — the Glue Data Catalog and schema drift: why a table has more than one schema at any moment, what a crawler changes when it runs, and why the partition that queries fine today fails tomorrow without anyone deploying anything.

AWS Architecture Series #60 — The one you cannot re-read