Week 22 - Container Image Security: A Signature Says Who, Not Whether

Why — The Problem This Solves

Someone on your team ships a new version of an app. It runs. A month later a question comes up in a review: who built that exact image, and did anyone check what was inside it?

Most teams cannot answer either. The image is just a tag in a registry. AWS now has a managed answer to both questions: ECR can sign every image as it arrives, and Amazon Inspector can scan it for known flaws. Both are close to one click.

Neither of them stops anything.

That is the whole of this week. Signing writes a signature. Scanning writes a finding. If nothing reads them and acts, a flawed image is signed, scanned, documented — and still deployable.

What You Need to Know — Skills & Tools

A container image is a packaged snapshot of a filesystem — OS libraries, language runtime and your code — built once and run identically everywhere. ECR stores them.

Amazon Inspector is a separate AWS service with its own bill. ECR offers "basic" and "enhanced" scanning as if they were two settings of one feature. Basic is ECR's own scanner, free. Enhanced hands the work to Inspector at $0.09 per first scan. You switch it on in the ECR console and the charges appear under a service you never opened.

The difference is worth the money for one reason: basic scanning reads OS packages only. Enhanced reads your language dependencies too — the pinned versions in requirements.txt or package-lock.json.

Architecture — How It Fits Together

Two managed features produce evidence. One function acts on it. Only the box on the right can stop a deployment, and it is the only box that is not a managed feature.

How We Built It — Step by Step

What we deployed

Twenty-four resources. Only four of them — the signing profile, the signing configuration, the Inspector enabler and the scanning configuration — exist because of signing and scanning. The rest is the enforcement nobody writes about.

Where we started

How we wrote it

One surprise shapes the whole file. ECR managed signing shipped in November 2025 and still has no resource in the Terraform AWS provider — the pull request adding it is still open, and 6.68.0 shipped on 7 October without it. Confirm it yourself rather than trusting me:

terraform validate
# Error: Invalid resource type ... does not support "aws_ecr_signing_configuration"

Cloud Control carries AWS::ECR::SigningConfiguration, so the feature is manageable declaratively today through the awscc provider:

resource "awscc_ecr_signing_configuration" "registry" {
  rules = [{
    signing_profile_arn = aws_signer_signing_profile.images.arn
    repository_filters  = [{ filter = "wk22-*", filter_type = "WILDCARD_MATCH" }]
  }]
}

Two traps sit in the schemas. Signing profile names accept [a-zA-Z0-9_] only, so wk22-signer is rejected and wk22_signer is not. And the scanning filter type is WILDCARD while the signing filter type is WILDCARD_MATCH — two strings for one idea.

How we deployed it

What appeared in the console

How we tested it

Three images, each isolating one variable: a clean current base, an end-of-life Ubuntu, and a current base carrying flask==2.0.0. All pinned by digest, because alpine:3.22 is whatever was published this morning.

They are built in CodeBuild, not on a laptop, and that is not convenience: signing follows whoever pushed, so a laptop push produces a signature attesting to a laptop — the exact thing signing exists to replace.

Challenges — What Actually Went Wrong

A valid signature on an image we refused to deploy

The headline test: an image with a complete, valid AWS Signer signature, quarantined anyway on 16 findings. Twelve of those were Python package findings — including CVE-2023-30861 on Flask — against four OS findings. Basic scanning would have reported the four and stopped.

A signature tells you who built an image. It tells you nothing about whether the image is safe to run. Those are different questions, and only one of them is answered by the feature with "signing" in the name.

The gate missed the case that matters most

Inspector emits its scan-complete event only the first time it sees a digest. One image rebuilt byte-identically, so the push attached a new tag to content Inspector already knew. No new scan, no event, no evaluation — and the image sat deployable with seven critical findings. Deleting it from ECR first did not help; Inspector still remembered.

This is not an edge case. Promoting an image by re-tagging it, v1.2.3 becomes prod, is how images normally reach production, and it pushes no new bytes. A gate that only reacts to first scans protects only images nobody has pushed before. The fix was a second trigger on ECR's own push event, deciding from findings Inspector already holds.

Three smaller ones, all found by tests

• The scan event's repository-name field holds an ARN with the image digest appended. The documented example shows no digest, so code written from the docs produces an invalid repository name and every quarantine fails.
• Quarantine removed only the tag named in the event. An image carrying two tags — the normal state of anything promoted — stayed pullable by the other one.
• Inspector reports a Python dependency's package manager as PYTHON, not PIP. A check looking for the wrong token does not fail loudly; it reports nothing found, which reads like a clean result.

Security — Controls at Every Layer

• Immutable tags. With mutable tags an image approved as v1.2.3 can be quietly repointed after review, and the signature and scan result then describe something that is no longer there.
• The gate quarantines, never deletes. Only the deployable tags go. Deleting the image would destroy the evidence needed to answer "what shipped, and when did we know".
• It fails closed, and alarms on its own errors. We saw this for real: when the gate broke, it raised rather than returning an allow.
• Least privilege. The build role needs signer:SignPayload as well as the ECR push actions. Without it the push succeeds and the image is simply never signed — no error anywhere.
• The signing key is not yours. A Signer profile takes a platform, a validity period and tags, and no key of any kind. For container signing AWS generates and holds the key material. You control who may ask it to sign; you never touch the key. That is a trust decision, not an absence of one.

Cost

Billed $0.2409. Almost all of it was signing.

I wrote, before the bill arrived, that signing was free. AWS Signer is free with ECR — its pricing page says so. The charge is on ECR’s side, as AsyncActions-ImageSigning, at $0.02 per signature, and ECR’s pricing page carries a "Managed Signing" heading with no figure under it. The bill was the only place that number appeared.

Twelve signatures, not three, because every rebuild re-signs. Four builds during one night of debugging cost four times a single build. On a busy pipeline that is the line to watch — not storage, which came to nine hundredths of a cent.

Two controls still matter more than the rates. Scanning and signing are scoped by repository filter, not registry-wide — and with signing billed per image, a registry-wide rule would sign everything you ever push. A lifecycle policy expires untagged images, because under continuous scanning every retained image is rescanned whenever the vulnerability database updates.

Cleanup

Inspector is enabled per account, not per stack — so it is worth being explicit about who removes it. Here aws_inspector2_enabler and the registry scanning configuration are Terraform resources, and terraform destroy did remove both: the teardown script found scanning already back to BASIC and all five Inspector scan types already DISABLED.

The script still checks, and still disables them if it finds them on. That is not superstition: in an earlier week a CloudWatch anomaly detector survived a destroy and kept billing for days, because nothing in the state file owned it. Account-level settings are the category where "the destroy probably handled it" is worth converting into a command that looks.

The script also cancels the signing profile. Signer profiles cannot be deleted, only cancelled — the documented end state rather than a leftover.

References

• ECR managed signing
• ECR enhanced scanning
• Inspector EventBridge event schema
• AWS Signer pricing
• The code for this week

Key Takeaways

• A signature says who, not whether. A valid signature on a vulnerable image is the most comfortable false assurance in this whole area.
• Signing and scanning stop nothing. They produce evidence. Something has to read it and act, and that something is yours to build and yours to get wrong.
• Enforcement keyed on "newly scanned" misses the normal case. Promoting an existing image pushes no new bytes and triggers no scan.
• Signing is billed per signature, and the rate is not on the pricing page. $0.02 each, 99.7% of this week’s bill, and every rebuild re-signs. Signer’s own pricing says "no additional charge", which is true of Signer and not of the feature.
• Test the enforcement, not the features. The managed parts worked on the first apply. The part that actually stops a bad image took five attempts, and every fault was found by a test rather than by reading the code.