Azure ML Workspace - Unable to get access token for ADLS Gen2

Question

Hello Microsoft Q&A, when running azure ml pipelines I got the following error: " permission denied when access stream. Reason: Some(This request is not authorized to perform this operation using this permission.) " When I checked the data assets for the pipeline, I got the follwoing error:

The Azure machine learning workspace is inside a vnet and I'm using a service principal for the data store authentification:

I allready granted the workspace managed identity AND the service principal the reader role for ALL private endpointes. Moreover I checked all other permissions, for example that the workspace managed identiy has the blob storage reader role for the adls gen2 storage. Does this has something to do with these changes: "Azure Machine Learning Network Isolation Changes with Compute Instance and Compute Cluster"

Could you please help me.

Answer 1

Hello @Lukas Thanks for reaching out to us. I haven't seen the same error, but based on some researches and my personal experience, the error message you're seeing indicates that the user or service principal running the Azure ML pipeline does not have sufficient permissions to access the data assets required by the pipeline. It's possible that the recent changes to Azure Machine Learning Network Isolation could be a factor in this issue.

Here are some steps you can take to further troubleshoot the issue:

Check the credentials being used to access the data assets: Verify that the credentials being used to access the data assets are correct and have sufficient permissions to read the data. You can check this by attempting to manually access the data assets using the same credentials and seeing if you encounter any issues.

Verify RBAC permissions: Make sure that the user or service principal running the pipeline has the necessary RBAC permissions to access the data assets. You can check this by reviewing the access policies and roles associated with the data assets, and making sure that the user or service principal is included in the appropriate role(s) with sufficient access.

Check firewall and network settings: If the data assets are hosted in a private network, make sure that the firewall and network settings allow the pipeline to access the data. You may need to configure virtual network peering or VPN connections to enable access.

Review pipeline configuration: Double-check the pipeline configuration to ensure that the correct data asset paths and permissions are specified. You can also try re-creating the pipeline from scratch to see if that resolves the issue.

Review the Network Isolation changes: Review the recent Azure Machine Learning Network Isolation changes and ensure that they are not impacting your pipeline. You may need to update your pipeline configuration to account for any changes in network isolation.

If none of the above steps resolve the issue, you can contact Microsoft support for further assistance. Please raise a support ticket if you have a support plan, please let us know if you have tried all above items but nothing works, I am happy to enable you a free ticket for this issue.

I hope this helps! Let me know if you have any further questions.

Regards, Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.

Answer 2

Hello @Yutong, thanks alot for your anwser. Unfortunately it is still not working. When running a pipeline in Azure ML, I get the following error:

Error Code: ScriptExecution.StreamAccess.Authentication
Native Error: error in streaming from input data sources
	StreamError(PermissionDenied(Some(NoIdentityOnCompute)))
=> permission denied when access stream. Reason: Some(NoIdentityOnCompute)
	PermissionDenied(Some(NoIdentityOnCompute))
Error Message: Authentication failed when trying to access the stream. Make sure you have correct permissions set up. Ok(NoIdentityOnCompute)| session_id=2c3f8189-ccdb-4003-850e-7246e702164c

My setting: In the pipeline, I want to read data from my adls gen2 storage. This storage is the default storage of my Azure Synapse workspace. The Azure Synapse workspace is inside a managed virtual network. The data exfiltration protection is enabled. I created a managed private endpoint inside Azure Synapse for my Azure ML workspace. The Azure ML worksapce is inside a VNET. The VNET has two subnets (training and scoring). Associated to the VNET, there is a network security group. Due to the Network Isolation changes, I added the following Outbound Security Rules:

After that, I recreated the compute cluster and submitted the pipeline again. Moreover I recreated the datastore and updated the secret of the service principal. I checked the permissions of the workspace managed identity and the service principal for ALL network ressources inside the ressource group. The managed identiy as well as the service principal both have at least the "Reader Role". They both have the "Storage Blob Data Reader" Role for the adls gen2 storage account. I'm using these private endpoints:

Here aml stands for Azure Machine Learning (you can ignore the pdre). So for example the first private endpoint connects the Azure Machine Learning workspace and the container registry. I would appreciate any help. King regards, Lukas

Answer 3

Hello @YutongTie-MSFT , we do not have a support plan, hence I would appreciate a free ticket for this issuse. King regards, Lukas