Using Service Principal (OID), Not Able to Access Azure Data Lake Storage from Azure Databricks Notebooks

Question

Hi All,

I am just mounting a directory of Azure Data Lake Gen2 instance in a Notebook cell using Service Principal. I fetched the Object ID (OID) of the Service Principal using the command "az ad sp show" and using the OID, I provided "Read", "Write" and "Execute" access to a directory in ADLS.

Now, from the Notebook in Databricks, I ran the commands to mount the directory, which ran successfully. Afterwards, I tried to run the command to see all the contents present in that monted directory using the command -
display(
dbutils.fs.ls("/mnt/datalakegen2oindrila/JSON-Data/")
)

but getting the following error every time -
AbfsRestOperationException: GET https://datalakegen2oindrila.dfs.core.windows.net/databricks-container?resource=filesystem&maxResults=500&directory=JSON-Data&timeout=90&recursive=false
StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
ErrorCode=AuthorizationPermissionMismatch
ErrorMessage=This request is not authorized to perform this operation using this permission.
RequestId:442104df-d01f-00bd-059d-de1943000000
Time:2020-12-30T11:18:29.2658260Z

Answer 1

Hello @Oindrila Chakraborty and welcome to Microsoft Q&A. Thank you for your question.

When using service principle for authentication, you should use the application (client ID), not the object ID.

Please give this a try.

Thank you for your patience,
Martin

Answer 2

@Oindrila Chakraborty when you granted access, did you give execute permissions, or just read/write?
In order to list the contents, a combination of both read and execute are required. With read and not execute, this error message happens.

Answer 3

@Oindrila Chakraborty , if we give any RBAC role to SP, would it not get access to all folders and containers of lake. i am trying to give my service principal access only to one container's folder. i am facing same issue. from storage explorer, i have added this SP and gave all ACL access, read/write/execute, still i am not able to access this folder from ADF.