Authentication issue when accessing the Azure Storage account from your Azure Databricks notebook

Question

I'm currently taking the Coursera Course called: "Perform data science with Azure Databricks" which is part of the "Microsoft Azure Data Scientist Associate Professional Certificate (DP-100)". Then I am in the Week 2, in the second lesson: "Read data in CSV format", and therefore I am using this notebooks:

https://github.com/solliancenet/microsoft-learning-paths-databricks-notebooks/blob/master/data-engineering/DBC/03-Reading-and-writing-data-in-Azure-Databricks.dbc?raw=true

The notebook called: "01-The-Databricks-Environment", has a cell that contains the next code:

files = dbutils.fs.ls("/mnt/training/")

for fileInfo in files:
  print(fileInfo.path)

print("-"*80)

In which at executed I got this issue:

last part of the output error description:

According Chat GPT the error message indicates that there is an authentication issue when accessing the Azure Storage account from your Azure Databricks notebook. This error commonly occurs when the authorization header in the request is not formed correctly.

Then, to troubleshoot this issue, Chat GPT recommended the following steps:

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Verify that you have provided the correct credentials for accessing the Azure Storage account. Double-check the storage account name and access key to ensure they are accurate.

Make sure that the Azure Storage account is accessible from your Azure Databricks workspace. Ensure that the networking configuration and firewall rules allow access to the storage account.

Check if the storage account key has expired or been regenerated. If so, update the access key in your notebook with the latest one.

Verify that you have the necessary permissions and roles assigned to your Azure Databricks workspace to access the Azure Storage account.

If you are using a service principal or managed identity for authentication, ensure that it has the appropriate permissions to access the storage account.

Try restarting the cluster in your Azure Databricks workspace. Sometimes, restarting the cluster can resolve temporary authentication issues.

If the issue persists after trying these steps, you may need to consult with your Azure Databricks administrator or Azure support for further assistance in diagnosing and resolving the authentication issue.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Then when I see the Access Key of the Storage Account of the Azure Databricks Workspace in which I'am the previously explained issue I got the next:

In the managed resource group of the Azure Databricks Workspace I got this:

Then inside the Storage Account resource I clic in the "Access Keys" section:

And I get this message:

The message says:

"The client 'live.com#carlos.alanis.leal@outlook.com' with object id 'a8d6a13b-57a9-4474-8a63-b039457ce679' has permission to perform action 'Microsoft.Storage/storageAccounts/listKeys/action' on scope '/subscriptions/16666c4b-1ba3-4952-abd0-d9fd395a369e/resourceGroups/databricks-rg-databricks_workspace_sam01-tvbptbnn5ts2w/providers/Microsoft.Storage/storageAccounts/dbstorageytzyskamibcmy'; however, the access is denied because of the deny assignment with name 'System deny assignment created by Azure Databricks /subscriptions/16666c4b-1ba3-4952-abd0-d9fd395a369e/resourceGroups/RG_Databricks/providers/Microsoft.Databricks/workspaces/databricks_workspace_sam01' and Id 'b085ec0b99bb4fd290008479c0de3859' at scope '/subscriptions/16666c4b-1ba3-4952-abd0-d9fd395a369e/resourceGroups/databricks-rg-databricks_workspace_sam01-tvbptbnn5ts2w'. "

So, I need to know if this issue is indeed the root cause of the error I encountered in the Notebook "01-The-Databricks-Environment" within the Azure Databricks Workspace, and how to fix it.

This issue is recurring in the subsequent notebooks of the course, so I urgently need to resolve it.

Additionally, several users of the course have reported the same issue on the Coursera forums. However, it appears that no solution has been provided yet, which is a significant problem.

Therefore, I kindly request assistance from the technical team at Microsoft. The Coursera course "Perform Data Science with Azure Databricks" is being taught by Microsoft, then I need help of the Microsoft team to resolving it.

Thank you.

Answer 1

have same issue, do you have any solution?