Error: This request is not authorized to perform this operation using this permission.", 403 in Azure Databricks notebook while running from ADF

Question

Hi ,
I see the below error when I run a databricks notebook from ADF.

Operation failed: "This request is not authorized to perform this operation using this permission.", 403, HEAD, https://*****datalakewe.dfs.core.windows.net/****/****/****/****.csv?upn=false&action=getStatus&timeout=90

Below is the code in notebook:
Trying to read a csv from azure data lake gen2.

Config_table=spark.read.format("csv").option("header",'True').load('abfss://****@****datalakewe.dfs.core.windows.net/*****/*****/)****.csv')

When I run the same cell manually in databricks, I face no errors.

I have been trying to understand if there is some access issue which is restricting data factory to run this.
Please provide your inputs on this.

Answer 1

Hello @Vinayaka Dalawai ,

Thanks for the question and using MS Q&A platform.

To resolve this issue, you need to verify the access permissions for the ADF and user type:

Note: Storage Blob Data Contributor: Use to grant read/write/delete permissions to Blob storage resources.

ADF permissions: Kindly check the permissions on the Storage account.

Check the user permission on the storage account.

For more details, refer to the below threads addressing similar issue:

• https://learn.microsoft.com/en-us/answers/questions/25915/azure-databricks-throwing-403-error.html
• https://learn.microsoft.com/en-us/answers/questions/775151/index.html

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Have you provided the access using Service Principal following the same steps as below:

https://deep.data.blog/2019/03/28/avoiding-error-403-request-not-authorized-when-accessing-adls-gen-2-from-azure-databricks-while-using-a-service-principal/