Business Challenge
A platform team runs 60 AWS accounts across four organisational units. Observability grew the way it usually does: the CloudWatch agent went onto everything because it was the default, log groups were created per service with whatever settings the module happened to have, and when the security team standardised on Splunk, subscription filters were added to forward everything there too. Later the SRE team brought in Datadog, and a second set of forwarders went in alongside.
Nobody designed this. Each step was reasonable. The result was an observability bill growing faster than the compute it was watching, and three separate places to look when something broke.
A log written by an application was ingested by CloudWatch Logs, then forwarded by a subscription filter and ingested again by Splunk, and in some accounts a third time by Datadog. Storage was then paid on all three. The team had never made a decision about this; the forwarders were added one at a time, each justified on its own.
FixDecide where a given signal is authoritative and route it there once, rather than copying everything everywhere.
FinOps flagged CloudWatch ingestion, and the obvious lever was the Infrequent Access log class, which has a lower ingestion price per GB. New log groups were created in that class. What nobody checked first: Infrequent Access does not support subscription filters. The logs kept arriving in CloudWatch and quietly stopped arriving in Splunk. It was found weeks later, during an investigation that needed those logs.
FixRead the feature matrix before changing log class. The cheaper class is a genuinely different product.
A log group's class cannot be changed after it is created. Fixing it meant creating replacement log groups in the Standard class, repointing every producer, migrating retention settings, and accepting that the historical data stayed behind in the old groups. A configuration mistake became a migration.
FixTreat log class as a create-time architectural decision, set explicitly in the module that creates the log group.
When the team evaluated moving some workloads off CloudWatch, they found the CloudWatch agent was installed and configured on every instance and container. Changing destination meant reinstrumenting the estate, so the evaluation stopped there. The bill was not negotiable because the architecture had no seam in it.
FixCollect through a vendor-neutral layer so that where telemetry goes is a configuration change, not a re-rollout.
Architecture
The framing that prevents all four is to stop treating observability as one choice. It is three, and they are independent: what collects the telemetry, how it is routed, and where it is stored. Teams conflate them, pick a vendor, and inherit all three answers at once.
Layer 1 — collection
The CloudWatch agent is the path of least resistance inside AWS and it is genuinely good at what it does. Its cost is that it speaks one destination. The alternative is ADOT, the AWS-supported distribution of the CNCF OpenTelemetry project. It handles traces, metrics and logs, offers auto-instrumentation, and collects AWS resource metadata so application performance can be correlated with the infrastructure underneath.
The architectural argument for ADOT is not that it is better at collection. It is that it exports to Amazon Managed Service for Prometheus, CloudWatch, X-Ray, OpenSearch, MSK and any OTLP-compliant backend. Instrument once, and the destination becomes a config change rather than a fleet-wide re-rollout. That is the seam the team in this scenario did not have.
Layer 2 — routing, and the console problem
Across 60 accounts the practical question is not only where data is stored but where an engineer looks. CloudWatch cross-account observability answers that without moving any data: designate a monitoring account, and source accounts share telemetry into it through a sink.
The monitoring account creates a sink
A sink is the attachment point. Each account gets one per Region, and the monitoring account declares which telemetry types it will accept.
Source accounts create links
Links are owned by the source account, which chooses what to share: metrics by namespace, log groups by filter, X-Ray traces, Application Signals services and SLOs.
Organizations makes it durable
Linking by organisation or OU means accounts created later are onboarded automatically — the difference between a design and a chore someone has to remember.
Scale is not the constraint
One monitoring account links up to 100,000 source accounts; a source account can share with up to five monitoring accounts, which is how security and platform teams get their own views.
If the monitoring account accepts more telemetry types than the source offers, the link succeeds and shares the intersection. If the source offers more than the monitoring account accepts, link creation fails outright. Widen the monitoring account first, then the sources.
Layer 3 — storage, where the money is
CloudWatch Logs offers two classes. Standard is fully featured. Infrequent Access has a lower ingestion price and is aimed at ad-hoc querying and after-the-fact forensics. The critical detail is what "lower price" actually covers: the classes differ in ingestion cost only. Storage and Logs Insights query charges are identical.
So Infrequent Access is a lever on ingestion, not a general discount — and it is paid for in capability.
Why This Architecture Holds Up
Cross-account observability is free where it matters most
For logs and metrics, cross-account sharing carries no additional charge, and the first trace copy is free. That makes the console layer close to a pure win: engineers get one place to search across 60 accounts without a second ingestion bill and without building a forwarding pipeline to a central account. Teams routinely build that pipeline anyway, because they assume centralising means copying.
The log class matrix is an architecture document
It is easy to read Infrequent Access as "same thing, cheaper". It is not. What it keeps: managed ingestion and storage, cross-account features, KMS encryption, most Logs Insights commands, S3 export, scheduled queries, sensitive data masking.
What it does not have is the part that catches people:
No subscription filters
The mechanism that forwards logs to Firehose, Lambda or a third-party SIEM. If Splunk or Datadog is downstream, this class breaks it.
No metric filters
No turning a log pattern into a CloudWatch metric, so any alarm built on log contents stops working.
No Live Tail or field indexing
Real-time troubleshooting and indexed lookups are gone, along with anomaly detection and embedded metric format.
No Container or Lambda Insights ingestion
And GetLogEvents/FilterLogEvents are unsupported — anything reading logs through the API has to move to Logs Insights.
Read as a whole, the split is coherent: Standard is for logs you operate on, Infrequent Access is for logs you keep and occasionally search. Debug logs from a chatty service, audit trails, and anything retained for compliance fit the second category well. Application error logs feeding alarms do not.
A log group's class is fixed once created. This belongs in the Terraform module that creates log groups, with a sensible default and an explicit override — not in a runbook someone consults later.
Portability is the cheapest insurance you can buy
The reason to collect through OpenTelemetry is not ideology about open standards. It is that observability contracts get renegotiated, vendors get acquired, and pricing models change. When they do, the question is whether changing destination is a configuration change or a project. ADOT makes it the former, and costs relatively little to adopt at the point where you are instrumenting anyway.
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Collection | ADOT for applications; CloudWatch agent for infrastructure | Applications are where lock-in hurts and where instrumentation is expensive to redo. Host metrics rarely leave AWS. |
| Central visibility | Cross-account observability, linked by Organizations | No extra charge for logs and metrics, no pipeline to build, and new accounts onboard themselves. |
| Log class | Standard by default; Infrequent Access deliberately | Set at creation and immutable. The wrong default becomes a migration rather than an edit. |
| Third-party forwarding | Route once, to whichever platform is authoritative | Copying every line to CloudWatch, Splunk and Datadog means paying ingestion three times for one event. |
| Long-term retention | Export to S3 | Storage charges are identical across log classes; S3 is the actual lever for anything kept for years. |
| Alarming on log content | Keep those groups in Standard | Metric filters do not exist in Infrequent Access, so log-derived alarms silently stop being possible. |
The question that orders the whole design
For each stream, ask what someone will actually do with it. Alarm on it, tail it live during an incident, query it during a post-mortem, or keep it because an auditor will ask in three years. Those four answers map cleanly onto Standard, Standard, Infrequent Access, and S3 — and they are business questions, not technical preferences.
List every log group with its class, retention, and whether it has a subscription filter. Nearly every estate finds groups retained forever that nobody queries, groups forwarded to two platforms for no reason, and debug logs sitting in Standard paying full ingestion. That inventory usually pays for the exercise on its own.
Closing Thought
Observability bills grow the way this one did: not through a bad decision, but through a series of individually reasonable additions that nobody ever looked at together. Another forwarder, another retention default, another agent. Each one small, none of them wrong on its own.
Splitting the problem into collection, routing and storage makes those additions visible as choices. It also makes the trade-offs legible: portability is bought at layer one, central visibility is nearly free at layer two, and almost all the money is spent at layer three. Teams that conflate the layers end up optimising the one they can see — usually storage — and breaking something in a layer they were not thinking about.
The honest caveat: none of this makes a third-party platform the wrong choice. Datadog and Splunk earn their cost when correlation across a heterogeneous estate matters more than the per-GB price, and a team that lives in one of those tools all day should not be forced into a second console for AWS alone. The mistake is not choosing them. It is paying for them and for the full-featured AWS-native path, for the same log line, without ever deciding which one is authoritative.
Edge — CloudFront, Origin Shield and WAF, and how the caching layer changes what your origin has to survive. Where the edge absorbs load, where it merely relocates it, and the configuration mistakes that turn a CDN into an expensive pass-through.
Official AWS Reference
- CloudWatch Logs log classes — the full feature matrix for Standard and Infrequent Access, the note that the classes differ in ingestion cost only, and the warning that a log group's class cannot be changed after creation.
Comments