Model training in local Azure ml Notebook

Hi everyone

I use a compute cluster of GPU for training a machine learning model.

For access we have this user managed identity to communicate with the Azure resources which correct configured with compute cluster.

The job for training model is created in the notebook. When submit the job it throws clientauthtication error.

ClientAuthenticationError: (UserError) Workspace nusaexrdeuseamlw MSI doesn't have appropriate permissions on the storage account nusaexrdeuseasta Code: UserError Message: Workspace nusaexrdeuseamlw MSI doesn't have appropriate permissions on the storage.

The storage account specified is the default one used while creating the Azure ML workspace.
Apart from storing metadata of job in default storage account we are not using it anywhere in the training model. Please help me out in solving this issue.

User's image

romungi-MSFT 44,771 Reputation points Microsoft Employee

2024-09-04T11:39:36.5433333+00:00

@Rajan,Naveen,US-St. Louis,External In case of compute cluster I think it can only support either of one system-assigned identity or multiple user-assigned identities, not both concurrently. See note from here.

If you are seeing this message, then I think it is using your user assigned identity as default since you have configured it.

The general behavior to select its default managed identity is the system-assigned managed identity or the first user-assigned managed identity. Since you have configured user-assigned managed identity it is being used and this identity might not have enough access for storage account. If this is the case, then at least Storage Blob Data Reader role needs to be added here. See this section for setting up access for jobs on compute cluster.

There are also some limitations here where this will not work if you are doing it via ML studio. Try using the same with SDK not from ML studio. Thanks!!
romungi-MSFT 44,771 Reputation points Microsoft Employee

2024-09-05T11:12:31.4233333+00:00

@Rajan,Naveen,US-St. Louis,External Did you get a chance to review the above suggestion and try the same by adding the Storage Blob Data Reader role with your user assigned identity?
romungi-MSFT 44,771 Reputation points Microsoft Employee

2024-09-06T11:37:22.6233333+00:00

@Rajan,Naveen,US-St. Louis,External did you get a chance to check if the above worked for you?
Rajan,Naveen,US-St. Louis,External 5 Reputation points

2024-09-09T05:52:13.4833333+00:00

Sorry I was not able see it past 2 days will check it out today.
Thanks romungi.
romungi-MSFT 44,771 Reputation points Microsoft Employee

2024-09-09T06:54:31.8466667+00:00

@Rajan,Naveen,US-St. Louis,External Please check and do let us know how does it goes.
Rajan,Naveen,US-St. Louis,External 5 Reputation points

2024-09-09T09:14:14.5666667+00:00

As You mentioned I just configured the only user managed identity used by the Compute cluster in job command setup This user managed identity is pointing to the namsprdigsoluseasta storage account.(Note: we are not storing any input or outputs in the datalake) The storage account specified here is not the default storage account configured with the workspace MSI which is nusaexrdeuseasta. The error now for job creation is

Encountered error when building image: ScriptTask failed with: Unable to upload blob to storage account because authorization failed This could be because the workspace's Managed Identity has been removed from the storage account. Please refer to https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-msi

Let me know if i configure the default storage account nusaexrdeuseasta in the user-managed identity could solve this issue.

Waiting for your valuable response.

Thanks
romungi-MSFT 44,771 Reputation points Microsoft Employee

2024-09-09T11:23:50.7033333+00:00

@Rajan,Naveen,US-St. Louis,External Yes, i think adding the default storage account in the MI and assigning it to the compute cluster will help. Also, the documented limitation of using it with CLI and SDK via the ML studio should be taken into account. Try to submit the job with the updated MI using SDK or CLI but not from ML studio.
romungi-MSFT 44,771 Reputation points Microsoft Employee

2024-09-11T07:26:38.5+00:00

@Rajan,Naveen,US-St. Louis,External Did you get a chance to test with the updated MI for this scenario? Thanks!!

Share via

Model training in local Azure ml Notebook

Your answer