AWS Daily Intelligence #23 - A Signing Key in Cleartext, and Why Upgrading the SDK Is Only Half the Fix

Executive summary

AWS published CVE-2026-83551 yesterday, in the Amazon SageMaker Python SDK. An HMAC secret key is stored in cleartext within pipeline definitions and accessible via the DescribePipeline API — which turns a read permission into a code execution permission for anyone else in the account.

Affected versions are v3 before 3.11.0 and v2 before 2.256.0. The fix is not a patched check: it replaces symmetric HMAC signing with asymmetric ECDSA signing, so the key that remains in the definition afterwards is a public key and reading it grants nothing.

The part worth stopping on is the remediation. Upgrading the SDK protects pipelines you create from now on. It does not touch the definitions you already have, and those still contain the cleartext symmetric key. AWS's own guidance includes a second step that is easy to read past: call pipeline.upsert() to replace stored symmetric keys with asymmetric public keys in existing pipeline definitions. A team that runs pip install --upgrade sagemaker, sees the version bump and closes the ticket has done half the work.

What changed

The interim control, for anyone who cannot upgrade immediately, is to restrict sagemaker:DescribePipeline IAM permissions to trusted users and use dedicated per-user S3 paths for isolation.

Architecture

To see why this is more interesting than an ordinary key-handling bug, it helps to know what the decorators actually do.

What is being signed, and why

The @remote decorator lets you run local Python as a SageMaker training job: the SageMaker Python SDK will automatically translate your existing workspace environment and any associated data processing code and datasets into a SageMaker training job that runs on the SageMaker training platform. That translation means serialising your function, its arguments and its dependencies, putting them in S3, and having the training job deserialise and execute them.

Deserialising a Python payload and running it is, by construction, arbitrary code execution. So the design signs the payload: the job verifies the signature before executing, and a payload that does not verify is rejected. That is a sound pattern and the right instinct.

Where it goes wrong

The signature was an HMAC, which is symmetric — the same key both signs and verifies. The training job needs the key in order to verify, so the key travelled with the pipeline definition. And the pipeline definition is readable through DescribePipeline.

That collapses the whole control. Anyone in the account who can call DescribePipeline can read the key, and with the key they can produce a payload that verifies. The bulletin describes the outcome directly: an attacker with account-level access to DescribePipeline could extract the key, forge malicious code payloads, and achieve unauthorized code execution within another user's pipeline context in the same AWS account.

Note who that attacker is. Not someone outside the account, and not an administrator. It is any principal holding what reads on paper as a describe-only permission — the sort of grant that goes into a read-only role for a data science team without a second look.

Why ECDSA rather than a patch

The fix replaces symmetric HMAC signing with asymmetric ECDSA signing, and that is the only repair available. With an asymmetric scheme the verifier needs the public key and nothing else, so the value that has to travel in the readable definition stops being a secret. The bug was not that the key was stored badly; it was that a symmetric scheme required a secret to be present wherever verification happened, and verification happened somewhere everyone could read.

That distinction is why the remediation has two steps. A patched check could be delivered entirely by a version bump. A change of cryptographic scheme has to rewrite the artefacts that carry the old scheme's key, and those artefacts are your existing pipeline definitions.

Business value

There is nothing to capture here, only exposure to close. The useful framing is what this tells you about a permission model you probably already have.

Most organisations grant read-only ML permissions liberally, on the reasonable theory that reading is safe. This is a concrete case where a describe permission on metadata conferred the ability to execute code as somebody else. If your account has a read-only data science role, the question worth asking today is not only whether the SDK is patched — it is which other describe permissions in that role return values rather than just descriptions.

Security considerations

Three things are worth separating.

• The blast radius is one account, and that is not reassuring. A multi-tenant ML platform where several teams share an account is exactly the shape this exploits: forging into another user's pipeline context is the whole point of the finding. Teams with per-team accounts are far less exposed, which is a decent argument for that account structure quite apart from this bulletin.
• The audit trail will look normal. A forged payload verifies, so the job runs as a legitimate pipeline execution. There is a DescribePipeline call in CloudTrail beforehand — indistinguishable from the thousands of describe calls a console session makes.
• Assume the key is compromised, not just exposed. Any principal who could read the definition could have taken a copy at any time, and the key is unchanged until you re-upsert. Rotation here is the upsert, which is another reason to treat that step as the fix rather than as tidying up.

Cost considerations

No cost to the upgrade itself. The re-upsert re-registers each pipeline definition, which is a control-plane call and not a training run, so it does not consume instance time.

The only cost worth planning for is the interim control, if you take it. Moving to dedicated per-user S3 paths means changing where the SDK writes its serialised payloads, and if you have lifecycle rules, bucket policies or cost allocation tags keyed to the old prefix layout, those follow. That is small, but it is the kind of small change that is easier to schedule than to discover mid-incident.

Operational considerations

The operational risk in this bulletin is not the vulnerability, it is the shape of the remediation. Version checks are automatable and universally deployed: dependency scanners, SBOM tooling and pip list all report the SDK version, and all of them will report success after the upgrade.

Nothing in that toolchain inspects a pipeline definition. So the closing condition for this ticket cannot be "the SDK is on 3.11.0" — it has to be "every pipeline definition in every account has been re-upserted since the upgrade", which nothing measures for you unless you build it.

Two practical notes. Pipelines created by people who have left, or by a notebook nobody runs any more, are still in the account and still carry the key — enumerate from ListPipelines rather than from what the team remembers building. And re-upserting on an old SDK version writes the symmetric key back, so the order matters: upgrade everywhere first, then upsert.

Tradeoffs

Implementation guidance

In order, because the order matters.

# v3
pip install --upgrade "sagemaker>=3.11.0"
# v2 line
pip install --upgrade "sagemaker>=2.256.0,<3"

# Do not forget the places that are not a laptop:
#   Studio images, notebook lifecycle configs, training containers,
#   CI runners, and any requirements.txt pinned to an older version.

Then enumerate rather than remember. Every pipeline in the account carries the old key until it is rewritten, including ones nobody has run in a year.

aws sagemaker list-pipelines \
  --query 'PipelineSummaries[].[PipelineName,LastModifiedTime]' \
  --output table

# For each pipeline, from code running the UPGRADED SDK:
#   pipeline.upsert(role_arn=...)
# Re-upserting on an old SDK writes the symmetric key straight back.

Finally, verify by looking at what the definition now contains rather than at the SDK version — aws sagemaker describe-pipeline --pipeline-name <name> returns the definition, and it is the one place the answer is unambiguous. If you are checking whether you were exposed rather than whether you are fixed, that same call is what an attacker would have made.

Best practices

• Never store a symmetric key in a structure your verifier reads from a shared location. If verification has to happen somewhere readable, the scheme has to be asymmetric. That is the general form of this bug and it recurs across systems that sign serialized payloads.
• Treat describe permissions on metadata as data access. A describe call that returns configuration returns whatever was put in the configuration, including things nobody intended to publish.
• Close remediation tickets on the artefact, not the version. Where a fix changes a stored format, the version check is necessary and not sufficient, and it is the check every tool performs automatically.
• Enumerate resources from the API when remediating. The set of pipelines, roles or buckets a team can name is reliably smaller than the set that exists.
• Subscribe to AWS security bulletins. This one appeared there and in no release note a data scientist would read.

Who should adopt this

Anyone using the SageMaker Python SDK's @step or @remote decorators should upgrade and re-upsert this week. If you use SageMaker only through the console, or build pipelines without those decorators, this specific component is not in your path — but check, because the decorators are the convenient way to do it and convenience spreads through a team without an announcement.

Treat it as urgent if you run a multi-tenant ML account where several teams or several people share one AWS account. That is the configuration the finding describes, and the one where a forged payload runs as somebody else.

Everyone else gets a free lesson: go and look at what your read-only ML role can describe.

Key takeaways

• Upgrade to 3.11.0 (v3) or 2.256.0 (v2), then call pipeline.upsert() on every existing pipeline. The upgrade alone leaves the cleartext key in definitions you already created.
• The attacker is a principal with a describe permission, inside your own account. sagemaker:DescribePipeline returned a symmetric signing key, so reading became executing.
• The fix is a scheme change, not a patch — symmetric HMAC to asymmetric ECDSA — which is precisely why stored artefacts have to be rewritten.
• Enumerate with ListPipelines. Pipelines nobody remembers are still holding the key.
• Interim controls exist: restrict sagemaker:DescribePipeline to trusted users, and give each user a dedicated S3 path. Neither removes the key.

Official AWS references

• AWS Security Bulletin — CVE-2026-83551, cleartext storage of HMAC signing key in the Amazon SageMaker Python SDK
• AWS Documentation — Run your local code as a SageMaker training job
• AWS Documentation — SageMaker Pipelines and the Python SDK
• AWS Security Bulletins