Azure ML Workspace - Unable to get access token for ADLS Gen2

Lukas 151 Reputation points
2023-04-11T15:48:31.0466667+00:00

Hello Microsoft Q&A, when running azure ml pipelines I got the following error: " permission denied when access stream. Reason: Some(This request is not authorized to perform this operation using this permission.) " When I checked the data assets for the pipeline, I got the follwoing error:User's image

The Azure machine learning workspace is inside a vnet and I'm using a service principal for the data store authentification: User's image

I allready granted the workspace managed identity AND the service principal the reader role for ALL private endpointes. Moreover I checked all other permissions, for example that the workspace managed identiy has the blob storage reader role for the adls gen2 storage. Does this has something to do with these changes: "Azure Machine Learning Network Isolation Changes with Compute Instance and Compute Cluster"

Could you please help me.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,611 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,473 questions
0 comments No comments
{count} votes

Accepted answer
  1. YutongTie-MSFT 46,996 Reputation points
    2023-04-12T04:14:47.7+00:00

    Hello @Lukas Thanks for reaching out to us. I haven't seen the same error, but based on some researches and my personal experience, the error message you're seeing indicates that the user or service principal running the Azure ML pipeline does not have sufficient permissions to access the data assets required by the pipeline. It's possible that the recent changes to Azure Machine Learning Network Isolation could be a factor in this issue.

    Here are some steps you can take to further troubleshoot the issue:

    Check the credentials being used to access the data assets: Verify that the credentials being used to access the data assets are correct and have sufficient permissions to read the data. You can check this by attempting to manually access the data assets using the same credentials and seeing if you encounter any issues.

    Verify RBAC permissions: Make sure that the user or service principal running the pipeline has the necessary RBAC permissions to access the data assets. You can check this by reviewing the access policies and roles associated with the data assets, and making sure that the user or service principal is included in the appropriate role(s) with sufficient access.

    Check firewall and network settings: If the data assets are hosted in a private network, make sure that the firewall and network settings allow the pipeline to access the data. You may need to configure virtual network peering or VPN connections to enable access.

    Review pipeline configuration: Double-check the pipeline configuration to ensure that the correct data asset paths and permissions are specified. You can also try re-creating the pipeline from scratch to see if that resolves the issue.

    Review the Network Isolation changes: Review the recent Azure Machine Learning Network Isolation changes and ensure that they are not impacting your pipeline. You may need to update your pipeline configuration to account for any changes in network isolation.

    If none of the above steps resolve the issue, you can contact Microsoft support for further assistance. Please raise a support ticket if you have a support plan, please let us know if you have tried all above items but nothing works, I am happy to enable you a free ticket for this issue.

    I hope this helps! Let me know if you have any further questions.

    Regards, Yutong

    -Please kindly accept the answer if you feel helpful to support the community, thanks a lot.

    0 comments No comments

2 additional answers

Sort by: Most helpful
  1. Lukas 151 Reputation points
    2023-04-12T14:32:40.3066667+00:00

    Hello @Yutong, thanks alot for your anwser. Unfortunately it is still not working. When running a pipeline in Azure ML, I get the following error:

    Error Code: ScriptExecution.StreamAccess.Authentication
    Native Error: error in streaming from input data sources
    	StreamError(PermissionDenied(Some(NoIdentityOnCompute)))
    => permission denied when access stream. Reason: Some(NoIdentityOnCompute)
    	PermissionDenied(Some(NoIdentityOnCompute))
    Error Message: Authentication failed when trying to access the stream. Make sure you have correct permissions set up. Ok(NoIdentityOnCompute)| session_id=2c3f8189-ccdb-4003-850e-7246e702164c
    

    My setting: In the pipeline, I want to read data from my adls gen2 storage. This storage is the default storage of my Azure Synapse workspace. The Azure Synapse workspace is inside a managed virtual network. The data exfiltration protection is enabled. I created a managed private endpoint inside Azure Synapse for my Azure ML workspace. The Azure ML worksapce is inside a VNET. The VNET has two subnets (training and scoring). Associated to the VNET, there is a network security group. Due to the Network Isolation changes, I added the following Outbound Security Rules: User's image

    After that, I recreated the compute cluster and submitted the pipeline again. Moreover I recreated the datastore and updated the secret of the service principal. I checked the permissions of the workspace managed identity and the service principal for ALL network ressources inside the ressource group. The managed identiy as well as the service principal both have at least the "Reader Role". They both have the "Storage Blob Data Reader" Role for the adls gen2 storage account. I'm using these private endpoints: User's image

    Here aml stands for Azure Machine Learning (you can ignore the pdre). So for example the first private endpoint connects the Azure Machine Learning workspace and the container registry. I would appreciate any help. King regards, Lukas


  2. Lukas 151 Reputation points
    2023-04-13T06:32:10.4466667+00:00

    Hello @YutongTie-MSFT , we do not have a support plan, hence I would appreciate a free ticket for this issuse. King regards, Lukas

    0 comments No comments