Databricks External Locations "Invalid Azure path" when listing files in ADLS container root

Question

I am running into an issue listing files in the root path of a container belonging to an ADLS Gen2 storage account on both Databricks runtime 11.2 and 10.4.

My workspace is connected to Unity Catalog and I created an external location called externalloc1 mapped to
abfss://******@storageaccount.dfs.core.windows.net

My user has "READ FILES" and "WRITE FILES" permission on externalloc1. The managed identity for Databricks has "Storage Blob Data Contributor" to the storage account.

I placed input folders and files in the root of the container. Running this command in a Python notebook on runtime 10.4 to list files and folders in the root raises an exception:

   %python  
   dbutils.fs.ls("abfss://******@storageaccount.dfs.core.windows.net/")  


   com.databricks.common.client.DatabricksServiceHttpClientException: INVALID_STATE: Invalid Azure path: abfss://******@storageaccount.dfs.core.windows.net  

Listing files in a folder works without issue:

   %python  
   dbutils.fs.ls("abfss://******@storageaccount.dfs.core.windows.net/data")  


   Out[4]: [FileInfo(path='abfss://******@storageaccount.dfs.core.windows.net/data/test.csv', name='test.csv', size=4, modificationTime=1664392087000)]  

Am I able to list files in the root of the container? I have seen references of other users listing files in the root of an external location on ADLS and S3 so I assume it would be possible.

Answer 1

Hello @NS97631-0872 ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is how to get the files and folder names on the root level of the container , please do let us know if its not accurate.

I did tried the setop on my side and it jkust works fine for me

I did tried out few things and it was behaving as expected . I started looking at the error merssage again "INVALID_STATE: Invalid Azure path:" , is it possible the name of the files/folder in the root is the culprit .
I suggest you to try a different ADLS gen2 account and create few folders or files and check .

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators