Week 20 - An Accountable Event Bus: What Happens to an Event Nobody Wanted

AWS Platform Engineering Lab · Week 20 of 52

24 September 2026 Terraform · HCP · EventBridge · CloudTrail no hourly meter · destroyed

📋 In This Post

Lab Overview
Cloud Services and Concepts
Architecture and Workflow
Prerequisites
Implementation
Validation and Evidence
Security and Cost
Troubleshooting and Lessons
Cleanup and Key Takeaways

Lab Overview

Picture a noticeboard. Anyone can pin a note to it. Separately, people leave standing instructions with the office manager — copy me on anything about orders. The person pinning the note never learns who read it, and that is the point.

The problem. Now someone pins a note reading order.crated. A transposed letter. No instruction matches it — and the board still accepts it and says "got it". Nobody ever reads that note, and nothing anywhere reports a problem, because the office only counts how many notes matched each instruction. Nobody counts the notes that matched nothing.

The objective. Build an event bus you can actually interrogate: who published, what was delivered, what matched nothing, and what happens to an event nobody wanted.

The outcome. PutEvents returning 200 means the bus accepted your event and nothing more. Three separate mechanisms were needed to see past that, and each one has a hole worth knowing about.

Difficulty	Intermediate
Lab runtime	Ran 24–26 September; no hourly meter in this build
Measured cost	$0.0000281 — no hourly meter at all
Deployment	Terraform via HCP Terraform with OIDC
Full lab guide	GitHub — prerequisites, Terraform, commands, validation, cleanup

Cloud Services and Concepts

Service	Purpose here	What it is
Amazon EventBridge	The bus itself — a custom bus, not `default`. Accepts published events and routes them by rule.	An event bus. Services and your own code publish events to it, and rules decide which events reach which targets.
EventBridge archive and replay	Keeps what was published (not what was delivered) and can re-publish it later.	An archive keeps a copy of what was PUBLISHED to a bus, not what was delivered, and replay pushes those stored events through the rules again.
Schema discovery	Infers event shapes from real traffic and writes them into a registry.	Watches real traffic on a bus and infers the shape of the events flowing through it, recording what producers actually send rather than what they were supposed to send.
AWS CloudTrail	Data events record who published, not what was delivered.	Records who called which AWS API, when, and from where. It answers who did it, not what the data contained.
AWS Lambda and Amazon SQS	The two subscribers, plus the queues that catch two different kinds of failure.	Lambda runs the code; SQS is a queue that holds messages until something collects them, including the ones that failed to deliver.
Amazon CloudWatch	Per-rule `MatchedEvents`, and the alarm on consumer errors.	AWS's metrics and logs service. Most services publish numbers into it automatically, without being asked to.

A 200 from PutEvents is an acceptance receipt, not a delivery receipt. It does not mean anything was listening, or that anything was delivered.

There is no metric for "matched no rule". You get MatchedEvents per rule and nothing for the gap between them. That gap has to be constructed — and the only way to construct it costs a second delivery of every event.

Architecture and Workflow

Every event lands somewhere: a rule, a queue, or the log group that proves nothing was dropped.

A publisher calls PutEvents against a custom bus and receives 200 plus an event id.
The orders rule matches on source and detail-type; matching events go to a Lambda consumer.
A catch-all rule matches everything and writes to a log group, so catch-all minus orders yields the count of events that matched nothing.
Delivery failures go to a rule-target DLQ; a consumer that accepts an event and then fails needs its own on-failure destination — two failure modes, two queues.
An archive retains what was published, independently of whether anything consumed it.
CloudTrail data events record who called PutEvents, scoped by ARN to this one bus.
Schema discovery infers shapes from the real traffic, including the mistakes.

Design decisions

A custom bus, not default. Scoping CloudTrail data events to one bus by ARN is only possible with a bus of your own — and an unscoped selector logs every PutEvents in the account and bills for all of it.

A catch-all rule purely to make a metric exist. This is the only way to detect unmatched events, and you pay for it with a second delivery of every event.

Two dead-letter paths, deliberately. They catch different failures, which most designs discover the hard way.

Prerequisites

Requirement	Detail
AWS account	Permission for EventBridge, Lambda, SQS, CloudTrail, CloudWatch and S3
Terraform	>= 1.10
HCP Terraform	Workspace with OIDC to AWS. No static keys
CloudTrail	Data events are off by default and billed per event — scope the selector by bus ARN
Region	`us-east-1`

No hourly meter in this build — no control plane, no NAT, no instance. The only standing charge is archive storage. That makes it unusually safe to leave running while you experiment, which is the opposite of the two weeks either side of it.

No prerequisite labs. Week 20 is self-contained.

Implementation

One custom bus, two subscribers and three watching mechanisms. The lab guide carries the full Terraform and the publish/replay scripts.

week-20-eventbridge-accountable-bus/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the modules
|   +-- modules/bus/        custom bus, both rules, archive, schema discovery
|   +-- modules/consumers/  Lambda consumer, both DLQs, the alarm
|   +-- modules/audit/      CloudTrail trail scoped to the bus ARN
|
+-- scripts/                publish, break a consumer, and replay

Only the directories that matter. The full repository is on GitHub.

The unmatched-event detector is arithmetic, not a feature:

catch-all MatchedEvents  -  orders MatchedEvents  =  matched nothing

There is no metric for it, so the catch-all rule exists purely to make one half of that subtraction exist.

EventBridge console showing week20-bus with schema discovery Started and four rules, two of them Managed — **Four rules, and I wrote two of them.** Turning on the archive and the discoverer silently installs a managed rule each — that is how those features consume from the bus. There is no privileged back channel; they subscribe like anything else. Every event is now evaluated against four rules, not two.

Validation and Evidence

Each mechanism was tested by publishing a real event and asking what could be seen afterwards.

Test	Expected	Actual	Status
Matching event reaches the consumer	Delivered and processed	Delivered and processed	Pass
Typo event (`order.crated`) is published	200, matched by nothing	200; detected only via catch-all arithmetic	Pass
CloudTrail attributes the publisher	Caller recorded	Recorded — who, not what	Pass
Rule-target DLQ catches a failing consumer	Message lands in DLQ	DLQ stayed empty — wrong failure mode	Fail — fixed with an on-failure destination
Consumer-error alarm fires	Notification sent	Fired and delivered	Pass
Replay re-delivers an archived event	Follows current rules	Delivered through a rule that did not exist when it was first published	Pass — see below

Test output showing eleven passed checks: a matched event delivered, an unmatched event returning 200 with nothing consuming it, the catch-all recording it, and the EventBridge DLQ staying empty while a Lambda on-failure destination fills — Eleven checks, nothing failed, and — after three of my own measurement bugs — nothing unmeasurable. `PutEvents` returned 200 and an event id for an event the consumer never saw.

The alarm notification as it arrives in an inbox: subject ALARM week20-bus-consumer-errors, state change OK to ALARM, with the account id and the subscriber address redacted — The only artefact that proves a human gets told. Its own description — *"the consumer raised, distinct from a delivery failure — this one arrived"* — is the finding in the next section, stated before the finding was known.

Replay test output showing an event published with no matching rule, a rule created afterwards, and the replay delivering the event through that new rule under a new event id — Publish an event nothing matches. Create a matching rule *afterwards*. Replay the window. It arrives — through a rule that did not exist when it was published, under a new event id. The console above records that a replay completed; only this shows what completing meant.

Security and Cost

Security

No static AWS credentials. HCP Terraform authenticates by OIDC.

The trail is scoped by ARN to one bus. An unscoped data-event selector logs every PutEvents in the account — including other projects' — and bills for all of it.

The DLQ's resource policy names the specific rule. Being a dead-letter queue grants a queue nothing; without the aws:SourceArn condition, any EventBridge rule in any account could write to it.

Log retention set explicitly. An implicitly created Lambda log group never expires, which is a standing charge nobody thinks to look for.

Attribution has a documented hole. When another account publishes to your bus through a resource policy, their account receives the CloudTrail data event and yours does not. Owning the bus does not mean seeing who wrote to it.

Cost

Billed $0.0000281, read from Cost Explorer once AWS posted the charges: CloudTrail data events $0.000013 and EventBridge $0.0000151, both on 24 September. The 25th and 26th billed zero with every resource still running — because there is no hourly meter in this build.

It is worth saying why I waited for the bill. The rate card would have let me publish a number a day earlier. This series once printed $1.40 for a week against a bill of $4.93 by doing exactly that. A calculation is not a measurement, and the gap between them is not always small.

The only standing charge is the archive, at $0.023/GB-month. Replay is billed at the same rate as publishing, $1.00 per million.

Troubleshooting and Lessons

The registry wrote the typo down as a real schema

Problem. Order.crated — published once, matched by nothing, consumed by nobody — acquired a full OpenAPI definition beside the real events.
Root cause. Schema discovery documents what producers actually send, and at the wire level there is no difference between a new event type and a bug.
Resolution. None available — discovery is not validation, and nothing rejected any of them. EventBridge accepted every event regardless of the registry's contents.
Lesson. The mistake does not merely escape detection; it gets written down as though it were intended, where the next engineer reads it as a contract.

EventBridge schema registry showing platform.orders@Order.crated, the typo, with a full OpenAPI 3.0 definition — `Order.crated` is the typo. Published once, matched by nothing, consumed by nobody — and now carrying a full OpenAPI definition beside the real event.

The dead-letter queue catches a different failure than you think

Problem. A DLQ was attached to the rule target, an event the consumer fails on was published, and the queue stayed empty.
Root cause. The EventBridge DLQ covers "I could not hand this over". A target that accepts an event and then fails has succeeded as far as the rule is concerned.
Resolution. A separate on-failure destination on the function itself — two failure modes, two queues.
Lesson. Check which half of the handover your DLQ actually guards.

Replay re-publishes into the present

Problem. A replayed event was delivered through a rule that did not exist when it was first published.
Root cause. Replay follows current rules, issues new event ids, and carries today's consequences.
Resolution. Treat replay as publishing, not as time travel.
Lesson. There is no way back to the original publisher, and no way to replay "as it was".

Cleanup and Key Takeaways

The sweep checks four things a "delete the bus" pass misses:

The archive is not deleted with the bus, and its storage keeps billing.
Discovered schemas outlive the discoverer. They live in the AWS-managed registry and cost nothing — which is exactly why they accumulate, including that typo.
The log-group resource policy is account-level and invisible in the log-group view.
Rules left on the default bus, which is where they land if event_bus_name is ever omitted from a rule or its target.

The replay test also creates a probe rule outside Terraform, so it removes it on exit via a trap. A script that creates AWS resources outside Terraform creates things the teardown cannot see.

Key takeaways

A 200 from PutEvents means the bus accepted your event. Nothing else. Not that anything was listening, not that anything was delivered.
There is no metric for "matched no rule". The detection has to be built, and the only way to build it costs a second delivery of every event.
A schema registry records mistakes as faithfully as intentions. Discovery is not validation.
Your dead-letter queue probably catches a different failure than you think. A target that accepts an event and then fails needs its own mechanism.
Replay re-publishes into the present — current rules, new event ids, today's consequences.

What happens to an event nobody wanted