Azure Architecture Series #16 — Remediation Tasks, and the Identity Nobody Regrants

Business Challenge

Post #15 was about the number an initiative reports. This one is about the gap between reporting a problem and fixing it, which in Azure Policy is not a matter of degree — it is a different mechanism with its own objects, its own permissions and its own failure modes.

The shape of the failure is consistent enough to be a genre. A platform team assigns a built-in that deploys diagnostic settings to every storage account. The definition is right. The assignment carries a managed identity, because the portal insisted on one. Compliance shows 400 non-compliant storage accounts on the day of assignment, and 400 non-compliant storage accounts six months later. Nobody is alarmed, because the number is not getting worse.

Nothing is broken. Every piece of that is documented behaviour:

• Existing resources are never fixed automatically. During an evaluation cycle, a deployIfNotExists definition marks matching resources as non-compliant and no action is taken on that resource. Only new and updated resources are remediated on their own.
• A remediation task is a manual object. It is created deliberately, it covers one policy at a time, and if the assignment is an initiative it must name the exact policyDefinitionReferenceId to remediate.
• The default task fixes 500 resources. That is 1% of the 50,000 maximum. A team that ran a task once, saw it succeed, and moved on may have remediated a twentieth of the estate.
• The identity's roles were granted once. The portal grants them when the assignment starts. If roleDefinitionIds is edited later — a new role added because the template grew — the new permission must be granted manually, even in the portal.
• A failed remediation does not look like a policy failure. The compliance state stays non-compliant, which is what it was before. The deployment errors are one page deeper.

Architecture

Of the eleven effects, exactly two change a resource: modify, which rewrites the request before the Resource Provider sees it, and deployIfNotExists, which deploys a template after the provider has returned success. Both require a managed identity on the assignment. Everything below follows from that division.

The two effects, and where each one intervenes

modify runs before the request is processed. Its operations are applied to the request content when the rule's if condition is met, and each operation can carry its own condition. It supports addOrReplace, Add and Remove, with Remove supported only for tags. Beyond tags it can set identity.type — but only on virtual machines and Virtual Machine Scale Sets — and can add or replace the values of aliases marked Modifiable.

Three of its skip conditions are worth knowing before trusting it as a control. When an operation's condition is false, that operation is skipped. When a property is not present in the request payload, modification may be skipped — a nested alias like ...deleteRetentionPolicy.enabled is not applied if the parent property is absent, because the omission is assumed to be intentional. And an attempt to set identity.type on anything that is not a VM or scale set skips evaluation altogether, making the resource not applicable, which is post #15's silent-denominator problem arriving through a different door.

conflictEffect decides which definition wins when two of them modify the same property, and its default is deny. For new or updated resources, a deny conflict effect takes precedence, and if more than one conflicting definition uses deny the request is denied outright. Microsoft's own recommendation is therefore explicit: definitions that include aliases should use the audit conflict effect, so a request on an API version where the property is not modifiable is allowed through with the operation skipped rather than failed. The default is the strict option; the documented advice is to change it.

deployIfNotExists runs after a configurable delay, once a Resource Provider has handled a create or update request and returned a success status code. A deployment occurs if there are no related resources, or if the resources found do not satisfy existenceCondition. Its details block is where most of the design lives:

Two constraints on the template itself catch people out at scale. Nested templates are supported; linked templates are not. And nested Microsoft.Resources/deployments should use unique names — the documented idiom is [concat('NestedDeploymentName-', uniqueString(deployment().name))] — to avoid contention between multiple policy evaluations running at once, which at estate scale is the normal case rather than the exception.

The remediation task: what it is and what it is not

A remediation task is a Resource Manager object of type Microsoft.PolicyInsights/remediations. It names one assignment, and for an initiative assignment one policyDefinitionReferenceId — the short name post #15 argued for setting deliberately, now doing real work, because a remediation can only remediate in the scope of one definition. Its settings and their defaults:

These settings cannot be changed after the remediation task begins. A task started with the defaults against a 12,000-resource estate is not adjustable mid-flight; it is cancellable, and then it is a new task. ProvisioningState takes one of Running, Canceled, Cancelling, Failed, Complete or Succeeded, and a DeploymentSummary reports total, successful and failed deployments. Note also that Azure Policy deletes remediation task resources 60 days after their last modification — so the task is a work order, not an audit record. If the evidence matters, capture it when the task completes.

The concurrency ceiling is generous enough to be dangerous. A maximum of 100 remediation tasks can run in parallel for a single policy definition or policy reference within an initiative, each with up to 30 parallel deployments — 100 × 30 = 3,000 concurrent deployments driven by one rule.

Access control: the chain, and the two links that never update

The identity model is the part that produces silent failure, and it is worth stating as a sequence.

1. The definition declares the roles. Under details, roleDefinitionIds lists the roles the template or the modify operations need. Built-ins arrive prepopulated. Custom definitions must declare them, with full resource identifiers.
2. The assignment carries the identity. Exactly one per assignment, system-assigned or user-assigned, though that one identity can hold multiple roles. For a user-assigned identity the scope where it is hosted need not match the assignment scope, but it must be in the same tenant. Its location does not affect how it operates.
3. Somebody grants the roles. Through the portal, Azure Policy grants the listed roles automatically once assignment starts. Through an SDK — which includes every pipeline, every Bicep deployment, every Terraform run — the roles must be granted manually, or the remediation deployment fails. A newly created identity must also finish replicating through Microsoft Entra ID before roles can be granted to it.
4. Somebody creates the task. And the deployment runs as the assignment's identity.

Two links in that chain are never revisited. Editing roleDefinitionIds in the definition does not regrant anything, in the portal or anywhere else. And three documented scenarios require manual grants even when the portal created the assignment: an assignment created through an SDK, a resource modified by the policy that sits outside the assignment scope, and a template that reads properties on resources outside the assignment scope. All three are ordinary designs — a diagnostic-settings policy writing to a central Log Analytics workspace in a platform subscription is exactly the second and third case at once.

Why This Architecture Holds Up

User-assigned identities, for a reason that only appears at the second change

The portal's default is a system-assigned identity, and for a single assignment that never changes it is fine. It stops being fine the moment an assignment has to be replaced — which post #15 established is not rare, because initiative-level parameters cannot be altered once assigned. A system-assigned identity is created with its assignment and dies with it, so every replacement produces a new principal with no role assignments, and every replacement therefore needs the grants redone. Miss it and the new assignment reports compliance perfectly and remediates nothing.

A user-assigned identity is a durable object with its own lifecycle. Its roles are granted once, to a principal that survives the assignment being recreated, and it can be created and permissioned by the same pipeline that deploys the definitions. It also makes the permission grant reviewable: the roles a policy holds over production become a resource in a repository rather than an invisible consequence of somebody clicking Save.

There is one thing to be careful about in exchange. One identity shared across several assignments accumulates the union of every role any of them needs, and the guidance is to restrict permissions to the smallest possible set. The right granularity is usually one identity per remediation domain — one for diagnostic settings, one for tagging, one for backup — not one per assignment and not one for everything.

Remediation is a deployment, so it needs the controls a deployment gets

A remediation task is not a report being refreshed. It is up to 3,000 concurrent template deployments against live resources, run by an identity with write permissions, from a task whose settings freeze the moment it starts. Everything an organisation would insist on for a change of that size applies: a bounded scope, a known blast radius, a failure threshold below 100%, an owner watching it, and a first run against something small.

The location filter is the only built-in narrowing available on the task itself, which means the practical way to stage a remediation is to create the task at a lower scope than the assignment — a single resource group, then a subscription — rather than to rely on filters. That also sidesteps the documented awkwardness at the top: for policies assigned on a management group, remediation tasks should be created after evaluation has determined resource compliance, not at assignment time.

The backlog is a project, and it is finite

There is a useful asymmetry hiding in all of this. Automatic remediation covers everything created or updated from the moment the assignment exists. The task-driven half covers only what already existed. So the remediation backlog is a fixed, shrinking set — it does not grow while you work through it, provided the assignment is enforcing.

That makes it a project with an end, which is worth saying out loud because the alternative reading — that remediation is an endless operational chore — is what causes teams to leave it undone. Size it once with a resourceCount that covers the estate, run it in scoped batches with a real failure threshold, and use ReEvaluateCompliance for the final pass so the last run is not acting on yesterday's evaluation.

Key Architecture Decisions

The shape that works

Closing Thought

The interesting thing about remediation is how cleanly it separates two claims that governance conversations routinely merge. We have a policy for that is a claim about a definition and an assignment. The estate complies is a claim about a set of deployments that somebody had to start, using an identity somebody had to permission, against a backlog that does not drain itself.

Azure Policy is honest about the difference. It marks existing resources non-compliant and takes no action, which is exactly the right default for a system that can deploy templates into production. The design work is in noticing that the default is a choice you are now responsible for: the automatic half covers the future, the manual half covers the past, and the past is the part with 400 storage accounts in it.

One test settles whether any of this is real in a given estate. Pick a remediating policy, open its assignment, find the identity, and check when its role assignments were last touched against when the definition was last changed. If the definition is newer, the policy has been reporting a control it cannot perform — and it has been doing so without a single error.

Official Azure Reference

• Details of the policy remediation task structure
• Remediate non-compliant resources with Azure Policy
• Azure Policy definitions deployIfNotExists effect
• Azure Policy definitions modify effect
• Details of the policy assignment structure
• Get policy compliance data