how to connect datalake gen2 storage to azure ml ?

Question

im trying to create a datastore in azure ml from a datalake gen2 storage , and i have registered an app and i gave it Storage Blob Data Reader permission but when i create the datastore i see it empty even the datastore has 3 data files. i did follow this https://www2.microstrategy.com/producthelp/Current/Gateway_Connections/WebHelp/Lang_1033/Content/adls_service_account_connectivity.htm
to register the app.

Answer 1

Sign in to Azure portal, select the correct directory, and create an app registration. During this process, you will get an Application (client) ID and Directory (tenant) ID which are important for later steps. Under your app registration, create a new client secret and save its value securely. This secret acts as the password for your application. Then navigate to your storage account, go to Access Control (IAM), and add a role assignment. Assign the "Storage Blob Data Contributor" role to the service principal you created. Store the client secret in Azure Key Vault for secure management. Then create Azure Key Vault-backed secret scope in Azure Databricks, where you'll provide details like the DNS Name and Resource ID of the Azure Key Vault. In your Azure Databricks workspace, use Python code to set up the connection to Azure Data Lake Storage Gen2. This involves using the secret scope and details like the application ID and tenant ID that you obtained earlier. If you have a firewall on Azure Data Lake Storage Gen2, ensure your Azure Databricks workspace can connect to it. This might involve setting up private endpoints or configuring virtual network service endpoints. Here is a step by step tuto : https://medium.com/geekculture/how-to-connect-azure-data-lake-gen-2-to-azure-machine-learning-510c00115add Connect to Azure Data Lake Storage Gen2 - Azure Databricks : https://learn.microsoft.com/en-us/azure/databricks/scenarios/databricks-datalake-gen2-get-started Azure Machine Learning : https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-upload-explore-data Azure Databricks & Spark - Azure Storage : https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-use-databricks-spark