Centralised Logging Platform: One Query Across Every Account

Why — The Problem This Solves

By Week 10 this lab's organization has real shape: a management account, a Dev OU with an active workload account, and Week 6's account vending machine ready to mint more. Every one of those accounts accumulates its own CloudWatch log groups — Lambda functions, ECS services, Step Functions executions — and they are all invisible to each other.

That creates two problems that get worse with every account added:

• Incident response scales linearly with account count. Answering "show me every ERROR across the platform in the last hour" means switching roles into each account, opening CloudWatch, and running the same Logs Insights query over and over. Ten accounts, ten logins, ten queries.
• Log evidence has the same blast radius as the incident. The only copy of an account's logs lives inside that account. If the account is compromised, misconfigured, or deleted — the exact scenarios where you need its logs most — the logs go with it.

Teams historically didn't fix this because the fix was genuinely expensive: an OpenSearch cluster, Kinesis Firehose plumbing from every account, index lifecycle management — easily $1,000+/month at enterprise scale and a part-time job to operate. That calculus changed in 2025: AWS made cross-account querying native and made the first centralized log copy free. The gap today is knowing the new primitives exist, not affording them.

What You Need to Know — Skills & Tools

Architecture — How It Fits Together

Two accounts, two complementary data paths. The dashed path is OAM: the source account shares its logs and metrics with the monitoring account — nothing is copied, nothing is billed. The solid path is the centralization rule: log events are physically replicated into the management account, so the evidence survives even if the source account doesn't. A scheduled Lambda in the source account generates realistic structured traffic so every layer has live data to prove itself.

How We Built It — Step by Step

Step 1 — Prerequisite: enable Organizations trusted access for CloudWatch

aws organizations enable-aws-service-access \
  --service-principal observabilityadmin.amazonaws.com

# CLI path only: the console would create this SLR for you
aws iam create-service-linked-role \
  --aws-service-name logs-centralization.observabilityadmin.amazonaws.com

# verify
aws organizations list-aws-service-access-for-organization

Step 2 — Terraform layout: one root, two providers, five modules

One HCP Terraform workspace (week-10-dev, VCS-driven) applies everything. The root declares two providers: the default one for the management account, and an aliased one that assumes OrganizationAccountAccessRole in the source account — so hub and spoke resources live in one plan with no static credentials anywhere.

terraform {
  required_version = ">= 1.10"
  required_providers {
    # centralization-rule resource exists only since provider 6.21.0 —
    # this week cannot use the 5.x pin earlier weeks carry
    aws = { source = "hashicorp/aws", version = "~> 6.39" }
  }
  cloud {
    organization = "Katta"
    workspaces { name = "week-10-dev" }
  }
}

provider "aws" {          # management: sink, centralized copy, alerting
  region = var.aws_region
}

provider "aws" {          # source: log generator + OAM link
  alias  = "source"
  region = var.aws_region
  assume_role {
    role_arn = "arn:aws:iam::${var.source_account_id}:role/${var.source_role_name}"
  }
}

Five modules, each owning one concern: oam_hub (sink + org-scoped sink policy), oam_spoke (the link, under the source provider), log_generator (Lambda + schedule + app log group), log_centralization (the org-wide rule), and alerting (destination log group, metric filter, alarm, SNS, dashboard).

Step 3 — Deploy via HCP Terraform

Push to main, HCP plans remotely, confirm the apply. 17 resources across both accounts from one run — the aliased provider's assume_role is exercised during the plan itself, so a broken cross-account role fails fast at plan time, not at apply time.

Step 4 — The OAM hub and spoke

The sink policy is the trust boundary: any account in the organization may link — so accounts minted by Week 6's vending machine join with zero config changes — but only for logs and metrics, enforced with a condition on oam:ResourceTypes.

resource "aws_oam_sink_policy" "this" {
  sink_identifier = aws_oam_sink.this.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = "*"
      Action    = ["oam:CreateLink", "oam:UpdateLink"]
      Resource  = "*"
      Condition = {
        StringEquals = { "aws:PrincipalOrgID" = var.organization_id }
        "ForAllValues:StringEquals" = {
          "oam:ResourceTypes" = ["AWS::CloudWatch::Metric", "AWS::Logs::LogGroup"]
        }
      }
    }]
  })
}

Step 5 — Live multi-account traffic

A platform demo with no data proves nothing, so the source account runs a generator: EventBridge invokes it every 5 minutes, and it emits weighted structured JSON — about 8% ERROR, 15% WARN, the rest INFO — across five fake services. Lambda is the right compute here for the same reason it keeps winning these one-shot jobs: millisecond duration, schedule-triggered, stateless, near-zero concurrency. (If this needed sustained hours-long load, an ECS task would be right; for high-throughput streaming, a Kinesis producer. Neither applies.)

Step 6 — The centralization rule

One Terraform resource defines the org-wide copy: source scope is the entire organization, and a LIKE filter on log group names keeps the rule surgical — only the platform's own /platform-lab/ groups are centralized, not every log group in every account.

resource "aws_observabilityadmin_centralization_rule_for_organization" "this" {
  rule_name = var.rule_name
  rule {
    source {
      regions = [var.source_region]
      scope   = "OrganizationId = '${var.organization_id}'"
      source_logs_configuration {
        encrypted_log_group_strategy = "SKIP"
        log_group_selection_criteria = "LogGroupName LIKE '${var.log_group_prefix}%'"
      }
    }
    destination {
      account = var.destination_account_id
      region  = var.destination_region
    }
  }
}

Step 7 — Analysis: one query, every account

Here's the part that makes the wait bearable: the "one query, every account" demo doesn't need the centralization rule at all — OAM sharing alone powers it. From the monitoring account, Logs Insights (now the "Log Analytics" experience, with the account scope set to All accounts) queries the source account's log groups in place:

SOURCE logGroups(namePrefix: ["/platform-lab"]) START=-10800s END=0s
| fields @timestamp, @aws.account, level, service, message
| filter level = "ERROR"
| stats count() as errors by @aws.account, service
| sort errors desc

The cross-account dashboard tells the same story in widget form — the right-hand widget reads the source account's Lambda metrics through OAM (note the xa cross-account badge):

Step 8 — Alerting: the platform pages someone

A logging platform that can't page anyone is a data lake. The metric filter turns ERROR-pattern events in the centralized stream into a number (ErrorCount in a custom namespace), the alarm watches it (≥5 in 5 minutes), and SNS emails the owner — including an OK notification when the spike clears.

One human step first: SNS email subscriptions must be confirmed from the recipient's inbox before any notification flows — the confirmation mail goes out when Terraform creates the subscription. With that clicked, and the centralized stream still blocked upstream, the alarm→SNS→email path was tested honestly but synthetically: injecting ErrorCount=12 directly into the metric with aws cloudwatch put-metric-data. The alarm tripped inside two minutes and the email arrived — proving everything downstream of the metric filter. The filter itself gets its live end-to-end proof when centralized events start flowing.

Challenges — What Actually Went Wrong

2"Centralization is currently initializing. Please try again later."

After enabling trusted access, CreateCentralizationRuleForOrganization refused with a ValidationException — and kept refusing, retried every few minutes for over 24 hours, while list/read APIs on the same service responded normally. Nothing in the docs mentions an initialization window at all. One suspect worth naming: this organization contains eight closed accounts sitting in their 90-day post-closure SUSPENDED state (decommissioned landing-zone leftovers), and an org-wide provisioning workflow that touches every member account may not enjoy meeting them.

It wasn't a CLI-only quirk either: the console's own rule wizard accepts the entire configuration and then rejects the final create with the same message — and the console's "Turn off trusted access" button greys out during initialization, so you can't even re-toggle it.

Fix (in progress at publish time)

Ruled out everything ruleable-out: the SLR exists, trusted access shows On, a delegated administrator was registered, CloudTrail confirms the calls land and get rejected. The remaining moves are a scripted retry probe (running), and an AWS re:Post escalation. The architectural lesson stands regardless of when AWS finishes: never put an org-wide enablement with an unbounded activation time on your critical path — this build originally chained its alerting module behind the rule via an unnecessary depends_on, which held six perfectly deployable resources hostage to the outage (Challenge 5). This section gets a dated update when the rule lands.

Security — Controls at Every Layer

• Org-scoped sink policy, resource-type limited. Only accounts inside the organization can link to the sink (aws:PrincipalOrgID condition), and only for logs and metrics (oam:ResourceTypes condition) — no account outside the org, no other telemetry types.
• Log evidence survives source-account compromise. Centralized copies live in an account the source account has no write access to — the core security argument for physical copies over pure sharing.
• No static credentials anywhere. HCP Terraform authenticates to AWS via OIDC; the spoke account is reached by role assumption from the management account; the SLR scopes what the centralization service itself can do.
• Least-privilege generator. The Lambda's role can write to exactly one log group — nothing else.
• Bounded retention as a control. 14 days at the source, 30 days centralized — logs are a liability as well as an asset; unbounded default retention is both a silent cost leak and a data-exposure surface.

Cost

The comparison that matters: the retired Firehose→OpenSearch pattern for a 50-account org at 100 GB/day runs $1,000–2,500/month before anyone queries anything. This design's equivalent: $0 first-copy ingestion, ~$90/month storage at 30-day retention, queries billed per GB scanned only when run.

Prices as of July 2026 — verify at aws.amazon.com/cloudwatch/pricing/.

Cleanup

The workspace is VCS-connected, so destroy runs in HCP: week-10-dev → Settings → Destruction and Deletion → Queue destroy plan. After the destroy, verify three things aren't still lingering: log groups under /platform-lab/week10 in both accounts, the OAM sink/link pair, and the centralization rule. Organizations trusted access for CloudWatch is an org-level setting, not a billed resource — leaving it enabled costs nothing and saves the initialization wait next time.

References

• AWS Docs — Cross-account cross-Region log centralization (CloudWatch Logs)
• AWS Docs — CloudWatch cross-account observability (OAM)
• AWS Blog — Simplifying Log Management using Amazon CloudWatch Logs Centralization
• AWS Blog — Amazon CloudWatch and Amazon OpenSearch Service integrated analytics experience
• AWS Solutions — Centralized Logging with OpenSearch (retiring December 2026)
• Terraform Registry — aws_observabilityadmin_centralization_rule_for_organization, aws_oam_sink, aws_oam_link

Key Takeaways

• Sharing and copying are different tools — run both. OAM answers "query everything now, for free"; centralization rules answer "keep evidence that survives the account that produced it." Neither replaces the other.
• The 2025 primitives deleted a whole architecture. Three billed hops (subscription filter → Firehose → OpenSearch) became one free native feature. Re-check "best practice" architectures against the current platform before rebuilding them.
• An irreversible, invisible choice: log class. Infrequent Access halves ingestion cost but permanently kills metric filters — decide per log group, at creation, with the alerting design in front of you.
• Terraform's graph only knows what you reference. The OAM link raced a resource policy it never referenced. When correctness depends on a policy existing, depends_on is the honest fix, not a retry loop.
• Brand-new AWS features have four clocks. Docs, console, CLI, and Terraform provider all described this feature slightly differently in the same week. Verify each layer live — including undocumented waits like the centralization backend's initialization.