The credential that has to work when nothing else does

Business Challenge

At 02:40 the corporate identity provider stops answering. Every engineer in the company is federated through it, so nobody can sign in to any AWS account. There is a production incident underway that needed hands twenty minutes ago.

The break-glass procedure exists. It is written down, it was reviewed, and it was signed off by two people who take this seriously. Step one is to retrieve the emergency credential from the enterprise password manager. The password manager authenticates through the identity provider that is down.

Someone remembers there is a root user. Its password is in the same vault. Its recovery email goes to a distribution list hosted on the corporate mail platform, which federates through the same directory. The registered phone number belongs to an engineer who left in March.

Nothing here was neglected. Every individual decision was the secure one — put the credential in the vault rather than a spreadsheet, use the corporate directory rather than personal accounts, put recovery on a distribution list rather than one person. Each of those is right in isolation, and together they built a break-glass procedure with a single point of failure that is also the most likely thing to fail.

The design question is not "who may use it"

Access control for break-glass is the easy half and it is where the attention goes. The hard half is a dependency audit: list everything the procedure touches, and for each one ask whether it still works in the scenario that made you reach for the procedure. A break-glass path that shares any link with the normal path is not a second path. It is the same path with a different label.

Architecture

Start by writing out both paths and looking for anything that appears in both.

A shared link is not a shortcut. It is the thing that will be broken on the day you reach for the glass.

The one place AWS endorses long-term credentials

Everything else in this series has pushed towards federation and temporary credentials. Break-glass is the documented exception, and AWS is specific about it. The general rule stands — we recommend you only use IAM users for use cases not supported by identity federation — and emergency access is named as one of those use cases:

In a situation where you can't access your identity provider and you must take action in your AWS account. Establishing emergency access IAM users can be part of your resiliency plan. We recommend that the emergency user credentials be tightly controlled and secured using multi-factor authentication (MFA).

That is worth reading as an architectural instruction rather than a concession. An IAM user with its own password and its own MFA device depends on nothing outside IAM. No SAML assertion, no directory, no permission set, no third party. It is the only identity in AWS with that property, and the property is the entire reason to accept credentials you would refuse anywhere else.

Two tiers, because they fail at different levels

Break-glass is not one thing. There are two, and conflating them produces either an unusable procedure or an over-privileged standing credential.

The emergency IAM user

For the common case: the identity provider is unavailable, or an Identity Center misconfiguration has removed everyone's assignments. This handles the overwhelming majority of real events, needs no root access, and can be scoped to a role rather than to administrator.

The root user

For the short list of tasks that genuinely require it, and for the case where IAM itself is the problem. AWS is blunt about the threshold: we strongly recommend you don't access the AWS account root user unless you have a task that requires root user credentials.

The circular dependency, stated by AWS itself

The scenario at the top of this post is not a hypothetical failure mode somebody imagined. It is documented, in one sentence, in AWS's own root user guidance:

Don't store the root user password with tools that depend on AWS services in an account that is accessed using that same password. If you lose or forget your root user password, you will not be able to access these tools.

The general form is broader than the sentence. Any credential whose purpose is to recover a system must be stored somewhere that does not depend on that system — and the dependency chain is usually longer than one hop. A vault that runs on EC2 in the account. A vault that authenticates through the IdP. A runbook in a wiki behind the same SSO. An alerting path that emails a mailbox on the same directory. Each is one link, and the procedure is only as available as its weakest one.

Splitting knowledge, in two places rather than one

A credential powerful enough to be worth breaking glass for is powerful enough that no single person should be able to use it alone. AWS suggests the split directly: consider using multi-person approval to make sure that no one person can access both MFA and password for the root user — one group holding the password, another holding the MFA device, one member of each required to sign in.

The part that gets missed is that the same split has to be applied to recovery, because recovery is a second way in. No one person should have access to both the email inbox and phone number since both are verification channels to recover your root user password. A perfect password-and-MFA split is undone by one administrator who can trigger a reset to an inbox they control and confirm it on a phone they hold.

Two other details make the root user survivable rather than theoretical. Use an email address that is managed by your business and forwards received messages directly to a group of users, and one that should not be used for other purposes — an individual's mailbox becomes a single point of failure the moment they leave. And you can register up to eight MFA devices on the root user, which turns "the person with the token is on a plane" from an incident into an inconvenience.

Why This Architecture Holds Up

It turns an argument about trust into a test anyone can run

"Is our break-glass good enough" is unanswerable and invites opinion. "List every system this procedure touches, and cross off the ones that survive an IdP outage" is a twenty-minute exercise with a definite result. The failure in the opening scenario would have been found by anybody who did it once.

The exception is bounded rather than open-ended

Accepting long-term credentials is a real cost, and the model keeps it small: one identity, one purpose, MFA required, alerted on use, tested on a schedule. That is a defensible position in front of an auditor precisely because the reasoning is written down — AWS names emergency access as a use case federation does not support, and the credential exists for that and nothing else.

Removing root from member accounts shrinks the problem enormously

For an organisation, the strongest move is to have fewer root users to protect. For multiple AWS accounts managed through AWS Organizations, AWS recommends removing root user credentials from member accounts, after which member accounts can't sign in to their root user or perform password recovery for their root user. Eighty accounts stop being eighty sets of credentials, eighty MFA devices and eighty recovery phone numbers to keep current.

MFA on root is no longer a recommendation to argue about

All AWS account types (standalone, management, and member accounts) require MFA to be configured for their root user, with registration required within 35 days of the first console sign-in attempt. The discussion about whether to enable it is over; what remains is the design question of who holds the device, and how many devices exist.

Key Architecture Decisions

1

Store the credential outside every system it can recover

This is the decision the whole design turns on, and the trap is that the secure-looking answer is the wrong one. An enterprise vault is the right home for almost every secret in the company and the wrong home for this one, if that vault authenticates through the identity provider or runs in the account.

The test to apply

Walk the chain and stop at the first thing that could be down: the vault, its authentication, the device you would read it on, the network you would read it over, the runbook that tells you the steps. Physical storage in a safe, split across two custodians, is not a regression to the 1990s here — it is the only medium with no dependency on anything you operate. Whatever you choose, the runbook has to be readable without signing in to anything.

2

Make it impossible to use quietly

A break-glass credential that can be used without anyone noticing is not a break-glass credential; it is a standing administrator credential with a dramatic name. The alert is not monitoring around the control — it is the control, because it is the only thing making the credential expensive to misuse.

Three layers, and the third is the one people skip

An EventBridge rule on the sign-in event into an SNS topic; GuardDuty, which has a finding type for root credential usage; and the part that is not technical. AWS says it directly: have procedures in place for how to respond to alerts so that personnel who receive a root user access alert understand how to validate that root user access is expected, and how to escalate if they believe that a security incident is in progress. An alert nobody knows how to judge at 3am is a notification, not a control.

3

Send the alert somewhere the outage cannot reach

The dependency rule applies to the alarm exactly as it applies to the credential, and this is the second-order mistake that survives the first review. An alert delivered to corporate email that federates through the failed IdP arrives nowhere. An alert into a chat platform behind the same SSO does the same.

Concrete rule

At least one alert path must terminate outside the estate: SNS to personal mobile numbers, or a pager service with its own authentication. AWS's guidance that alerts should include, but not be limited to, the email address for the root user is pointing at the same thing — the root mailbox is one channel and must not be the only one, since it is also a recovery channel and therefore part of what an attacker would target first.

4

Test it on a schedule, and expect the compliance dashboard to go red

An untested break-glass credential has an unknown state. Passwords expire, MFA devices desynchronise, custodians leave, phone numbers get reassigned, and none of that produces a signal. The test is not "does the credential exist" but a full rehearsal: two custodians, retrieve, sign in, confirm the alert fired and reached a human, sign out, record it.

The finding that is not a finding

If you follow the advice to remove root credentials from member accounts, be ready for this: MFA-related rules return noncompliant if you follow the best practice of removing root user credentials, until those accounts are evaluated as not applicable. Doing the right thing turns the dashboard red first. Know that before someone opens a ticket about it.

The dependency audit, as a table

What the procedure touches	Survives an IdP outage?	What to do instead
Enterprise vault behind SSO	No	Offline storage, split across two custodians
Vault hosted in the account it protects	No	Anywhere with no dependency on that account
Runbook in a wiki behind the same SSO	No	Printed, or in the safe with the credential
Root recovery email on the corporate directory	Usually not	A group address that does not depend on the directory
Alert to corporate email or chat	No	SNS to mobile numbers, or an external pager
Emergency IAM user with its own MFA	Yes	This is the design
Root user, password and MFA split	Yes	Reserve for root-only tasks

One thing not to do, however tempting

Do not create access keys for the root user. AWS's wording is unambiguous — we strongly recommend that you do not create access keys for your root user because the root user has full access to all AWS services and resources in the account, including billing information. A key exists to be used unattended, which is the exact opposite of the property break-glass needs. Break-glass should be a console sign-in that two people perform deliberately and that sets off an alarm.

Reference: the alert that makes it loud

An EventBridge rule on the console sign-in event, filtered to the root user. This is deliberately noisy: every root sign-in fires it, including your own scheduled test, which is how you learn the path still works.

EventBridge — every root sign-in, no exceptions

{
  "source": ["aws.signin"],
  "detail-type": ["AWS Console Sign In via CloudTrail"],
  "detail": {
    "userIdentity": {
      "type": ["Root"]
    }
  }
}

Two notes on wiring it up. CloudTrail distinguishes ordinary root sign-in from a privileged root session — you can use the sign-in event to identify the actions taken by the management account or a delegated administrator using sts:AssumeRoot — so a rule matching only console sign-in will miss centrally-initiated privileged tasks, and both belong in the alert. And the SNS topic this publishes to should have at least one subscriber that is not an address on your corporate mail platform, for the reason in decision 3.

Closing Thought

Break-glass gets designed as an access control problem because that is the part with a clear answer: restrict who holds it, require MFA, log the use. All of that is necessary and none of it is the hard part. The hard part is that the procedure has to work in a world where the things you normally rely on are the things that have failed, and every convenient choice quietly reintroduces one of them.

This is also the post in the series where the advice runs against everything before it. Federate, use temporary credentials, never store long-lived secrets — and then, for exactly one identity, do the opposite deliberately, because the property you need is independence and nothing federated has it.

The test is one sentence: describe the outage that makes you reach for this, then walk the procedure and stop at the first step that outage breaks. If you get to the end, you have a break-glass procedure. If you stop at step one because the password manager is behind the identity provider, you have a document. Most organisations have a document, and find out which they have at 02:40.

Next in this series

Security & Identity — the security tooling account: what belongs in it, why the detective services are delegated there rather than run from the management account, and the difference between an account that holds findings and one that can act on them.

AWS Architecture Series #42 — The credential that has to work when nothing else does