Azure Architecture Series #36 — Managed Identity on VMs, App Service, Functions and AKS

Business Challenge

Post #35 argued for managed identities on their merits: no credential, no licence, no cost, and a user-assigned identity that behaves as the unit of authorisation for everything that runs. That post was about the object. This one is about the plumbing, because "enable managed identity" means something materially different on each host, and the differences are exactly the kind that survive code review and fail in production.

The short version: a VM asks a shared IP address for a token. An App Service app asks a private URL supplied to it in an environment variable. A pod in AKS asks nobody — it reads a file its cluster wrote, and trades it with Microsoft Entra ID for a token. Three mechanisms, one feature name.

Almost all of that is invisible if you use the Azure Identity client libraries, which is why they are the right answer and why this post ends up recommending them. But "use the SDK" is advice you can only follow deliberately if you know what it is hiding, and three things underneath are architecture decisions rather than implementation details: who can use the identity, what happens when you do not name it, and what AKS is actually doing instead.

Architecture

Virtual machines: a link-local address and one mandatory header

The VM path is the oldest and the most literal. A token comes from the Azure Instance Metadata Service:

GET http://169.254.169.254/metadata/identity/oauth2/token
      ?api-version=2018-02-01&resource=https://management.azure.com/
Metadata: true

Two details carry weight. The Metadata: true header is not decoration — it is used as a mitigation against server side request forgery (SSRF) attacks, and must be set to "true", in all lower case. The reasoning is worth following: 169.254.169.254 is reachable from any code on the box, including code that has tricked your application into making an HTTP request on its behalf. Requiring a header that a naive URL-fetching bug will not set is the cheap defence against that.

The second: with more than one user-assigned identity attached, object_id, client_id and msi_res_id become required, if your VM has multiple user-assigned managed identities. Ambiguity is an error here, which turns out to be the kinder behaviour — see App Service below.

The operational surface is unusually well specified, and worth encoding once in whatever wrapper you use:

• Retry 404, 429 and 5xx. Do not retry 4xx, which are design-time errors.
• 410 is a scheduled outage with a stated bound: IMDS is going through updates and will be available within 70 seconds.
• 429 is throttling — throttling limits apply to the number of calls made to the IMDS endpoint. Cache tokens; the subsystem caches too, but we still recommend that you implement token caching in your code.
• Back off exponentially: five attempts at roughly 0, 2, 6, 14 and 30 seconds.
• Never branch on the error text. Error descriptions can change at any time. Do not write code that branches based on values in the error description.
• No proxies. IMDS isn't intended to be used behind a proxy and doing so is unsupported — which catches estates that set a blanket HTTP_PROXY across every VM.

App Service and Functions: a private endpoint, and a different word for the same thing

There is no IMDS here. Instead the platform makes this endpoint available by defining two environment variables: IDENTITY_ENDPOINT, the URL to the local token service, and IDENTITY_HEADER, a header that can help mitigate server-side request forgery (SSRF) attacks whose value the platform rotates.

That is a better design than the VM's, because the secret is per-app and rotating rather than a well-known constant. It is also completely incompatible, and the incompatibilities are petty enough to be dangerous:

msi_res_id against mi_res_id. One letter, same meaning, different host. Nothing about moving a workload from a VM to App Service will surface that until the token request comes back wrong — and on App Service, "wrong" has a specific and unhelpful shape.

Two more App Service specifics that catch people. Identity is per-slot: the managed identity configuration is specific to the slot, so a staging slot has its own identity and its own grants, and a swap does not carry them across. And identities do not survive a tenant move: because managed identities don't support cross-directory scenarios, they don't behave as expected if your app is migrated across subscriptions or tenants, after which downstream resources also need to have access policies updated to use the new identity.

One anti-feature worth knowing exists so you can recognise it: WEBSITE_DISABLE_MSI disables only the local token service but leaves the identity in place, and tooling still shows the managed identity as on or enabled. Microsoft's own verdict — we don't recommend that you use this setting. If you meet an app whose identity is configured, visible, and returning nothing, this is the setting to look for.

AKS: the same identity, reached the other way round

The third host is not a variation on the first two. The identity at the far end is still usually a user-assigned managed identity — federated identity credentials are created on one, which is why the limit below is expressed per managed identity. What is missing is the endpoint: nothing local hands the pod a token. A Kubernetes token is issued, and exchanged for an Entra one.

Microsoft Entra Workload ID uses Service Account Token Volume Projection to enable pods to use a Kubernetes identity, and then OpenID Connect (OIDC) federation enables Kubernetes applications to access Azure resources securely with Microsoft Entra ID, based on annotated service accounts. The inversion is stated directly: the AKS cluster acts as the token issuer. Entra uses OIDC to discover public signing keys and verify the authenticity of the service account token before exchanging it for a Microsoft Entra token.

So the trust flows the other way. Kubernetes mints a token, Entra has been told to trust that issuer, and the exchange happens at Entra's v2 endpoint rather than at any Azure-local address. Which produces a failure mode unique to this host: a raw resource URI, such as https://management.azure.com/, can fail because workload identity uses the Microsoft Entra v2 token endpoint rather than the IMDS resource flow used by managed identity. Scopes must be <resource>/.default. Code carried over from a VM will pass the wrong shape and get an error that says nothing about hosts.

Three more that decide whether a deployment works:

• The label is load-bearing. azure.workload.identity/use: "true" is required in the pod template spec, because only pods with this label are mutated by the azure-workload-identity mutating admission webhook. Without it, the pods fail after they're restarted — so a deployment can look healthy right up until the first restart.
• Never hard-code the token path. The webhook sets the AZURE_FEDERATED_TOKEN_FILE environment variable to the path of the token file, and the guidance is explicit: Don't hard-code a path ... The mount path is an implementation detail of the webhook and can change.
• Re-read the file. Kubernetes refreshes the projected token in place before it expires, so read it again each time you exchange the token rather than caching its contents for the lifetime of the process.

And a limit that shapes multi-cluster designs: a maximum of 20 federated identity credentials per managed identity. One credential per cluster means one identity stops at twenty clusters — which is why identity bindings exist in preview, and why post #37 takes federation on its own terms.

Why This Architecture Holds Up

Because the boundary is the host, not the process

Return to that sentence: all code/scripts running on a virtual machine can request and retrieve tokens for any managed identities available on it.

There is no process isolation in that statement. A sidecar, a monitoring agent, a scheduled task, an SSH session, a compromised dependency — each can call 169.254.169.254 and get the same tokens your application gets. Attaching two user-assigned identities to one VM does not give you two compartments; it gives every process on the box both sets of permissions.

The design consequence is concrete. If two workloads need different permissions, that is an argument for two hosts, not two identities on one host. And it reframes the containers question: on AKS, workload identity is per-pod via the service account, which is a genuinely finer boundary than the VM gives you — one of the better reasons to prefer it that has nothing to do with Kubernetes fashion.

Because "it works on my VM" is not portable

Every row of that comparison table is a migration hazard, and none produces a helpful error. Move a service from a VM to App Service and the endpoint, the header, the API version and the parameter spelling all change. Move it into AKS and the whole model changes, including the scope format.

Which is the real argument for the Azure Identity libraries, and it is stronger than convenience. They read IDENTITY_ENDPOINT when it exists, fall back to IMDS when it does not, and read AZURE_FEDERATED_TOKEN_FILE in a pod — the documentation confirms they read AZURE_FEDERATED_TOKEN_FILE for you. Hand-rolled HTTP calls to a token endpoint are host-specific code that looks host-agnostic, and that is the trap. If you must call the endpoint directly, the wrapper belongs in one place with the retry table above encoded in it.

Because the silent substitution deserves a rule of its own

Of everything here, the App Service fallback is the one to design against, because it is the only failure that produces a working token for an identity you did not ask for.

The rule that removes it: always name the identity explicitly, even when there is only one. Passing client_id costs nothing when the app has a single identity, and it converts a future silent substitution — the day someone enables the system-assigned identity for an unrelated reason — into an error at the right place. This is the same discipline as post #32's advice to create groups role-assignable on day one: the cheap explicit choice now removes an expensive implicit one later.

Because the AKS token lifetimes are two clocks, not one

A detail easy to miss and awkward to debug: Kubernetes service account token expiry isn't correlated with Microsoft Entra tokens. Microsoft Entra tokens expire in 24 hours after they're issued. The projected service account token defaults to 3600 seconds with a supported range of 3600-86400.

So there are two independent expiries in play: the Kubernetes token, refreshed in place by the kubelet, and the Entra token obtained by exchanging it. A process that caches the file contents rather than re-reading them will work for an hour and then fail at a boundary that has nothing to do with its own logic. That is precisely why the guidance says to re-read the file each time.

And when permissions change, post #35's cache still applies underneath all of this — the back-end maintains a cache per resource URI for around 24 hours, and it cannot be flushed. Three timers, none of which you control.

Key Architecture Decisions

Closing Thought

"Enable managed identity" is a single sentence in an architecture document and three unrelated mechanisms underneath it. A link-local address guarded by a header. A private URL guarded by a rotating secret. A file written by a cluster and exchanged with a directory that has been told to trust it. The only thing they truly share is the promise: no credential in your code.

What strikes me, having read all three, is how differently they fail. The VM refuses an ambiguous request. App Service answers it with the wrong identity. AKS works perfectly until a pod restarts. Those are three different relationships with the operator — strict, accommodating, and deferred — and only the first is the one you would choose for a security mechanism.

Which is why the two rules in this post are worth more than the endpoint details: always name the identity, and let the SDK find the door. The first removes the failure that returns a valid token for the wrong principal. The second means the next migration changes where your code runs and not what it says.

Official Azure Reference

• Use managed identities on a virtual machine to acquire an access token
• Managed identities in Azure App Service
• Use a Microsoft Entra Workload ID on AKS
• Azure services that support managed identities
• Best practice recommendations for managed identities