I am getting an error: "Endpoint identity does not have pull permission on the registry." while deploying a custom trained model in azure.

Question

ResourceOperationFailure: Endpoint identity does not have pull permission on the registry. For System Assigned Identity case and workspace ACR, platform will wire up the permissions but if user is creating deployment with User Assigned Identity or if using non-workspace ACR, then User should grant pull permissions to identity on the registry.

I am trying to deploy a finetuned (RoBERTa base model) in azure.
The model is stored as safetensors.

Steps followed:

1. Created a separate Azure container registry which can be used.
2. Created a separate managed identity resource (uami).
3. Created a azure machine learning workspace (with new key vault, blob storage).
4. Provided AcrPull permission to the workspace and the uami from the acr container resource.
5. Provided Storage Blob Data Reader permission to the uami from the blob storage.
   I have creating a score python script (Inference_PN.py) to be used for deployment. This has an init() and run(raw data) functions.

Also I have created a conda dependencies yml file which contains the necessary python libraries to be installed in an environment.

In the notebook environment, I have uploaded the finetuned models stored as safetensors which would be used for deployment (The model was trained in a separate workstation).

from azure.identity import DefaultAzureCredential
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Model
from azure.ai.ml.entities import Environment
from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment, IdentityConfiguration
from azure.identity import AzureCliCredential
from azure.ai.ml.entities._credentials import ManagedIdentityConfiguration
from azureml.core import Workspace


ws = Workspace.from_config()


credential = DefaultAzureCredential()

ml_client = MLClient(
    credential=credential,
    subscription_id=ws.subscription_id,
    resource_group_name=ws.resource_group,
    workspace_name=ws.name
)


# Model registration
model = Model(
    path="./models/PN",
    name="mm_roberta_pn_v1",
    description="RoBERTa model with safetensors",
    type="custom_model"
)
registered_model = ml_client.models.create_or_update(model)


# Environment creation
env = Environment(
    name="roberta-pn-env",
    image="mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest",
    conda_file="./CondaDependencies/PN_dependencies.yaml"
)
env = ml_client.environments.create_or_update(env)

# Endpoint creation with the uami resource
uami_resource_id = "<my-uami-resource-id>"
endpoint = ManagedOnlineEndpoint(
    name="roberta-pn-endpoint-v2",
    auth_mode="key",
    identity=IdentityConfiguration(
        type="user_assigned",
        user_assigned_identities=[
            ManagedIdentityConfiguration(resource_id=uami_resource_id)
        ]
    )
)
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

#======= This is where the code fails with above error ===========================
# Deployment
deployment = ManagedOnlineDeployment(
    name="roberta-pn-deployment", 
    endpoint_name=endpoint.name,
    model=registered_model.id,
    environment=env.id,
    code_path=".",
    scoring_script="Inference_PN.py",
    instance_type="Standard_DS3_v2",
    instance_count=1
)
ml_client.online_deployments.begin_create_or_update(deployment).result()
#====================================================================


ml_client.online_endpoints.begin_update(
    endpoint_name=endpoint.name,
    default_deployment_name="roberta-pn-deployment"
).result()

The code throws an error with the above error message in the block highlighted.
I have ensured the identity of endpoint being a user_assigned type.

for the endpoint this is the outcome for its identity configuration:

{
  "tenant_id": <>,
  "type": "user_assigned",
  "user_assigned_identities": [
    {
      "client_id": <>,
      "principal_id": <>,
      "resource_id": <>
    }
  ]

I have also tried not to use the user assigned identity and use the system assigned default identity in which case the endpoint identity shows as system assigned type. The error persists. Need some help how to resolve this issue.

Answer 1

Hi Mrinmoy,

Thank you for contacting Q&A Forum.

To resolve the issue with the user-assigned managed identity not having pull permissions on the registry, you need to ensure that the managed identity is correctly assigned to the deployment. Please refer to the instructions provided in the Azure Machine Learning documentation here to define the deployment configuration.

Link: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-resources-from-endpoints-managed-identities?view=azureml-api-2&tabs=user-identity-python#define-the-deployment-configuration

If I have answered your question, please accept this answer as a token of appreciation and don't forget to give a thumbs up for "Was it helpful"!

Best Regards,