FileNotFoundException when using abfss to list files in Azure Databricks!

Goel, Akanksha 66 Reputation points
2020-08-04T18:22:18.177+00:00

Hi team,

I am trying to connect to ADLS2 using hadoop configurations:
15557-image.png

But when I am trying to use FS commands to list all the files on the path, i am getting File not found exception:

import org.apache.hadoop.fs.{FileSystem, Path}
val fs = FileSystem.get(sparkSession.sparkContext.hadoopConfiguration)
val listOfFiles = fs.listFiles(new Path("abfss://conhw@saditdhprocessed.dfs.core.windows.net/clickstream/driverenum/temp"), false)

I am getting this error:
15558-image.png

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,340 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,917 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 77,081 Reputation points Microsoft Employee
    2020-08-05T10:09:52.7+00:00

    Hello @Goel, Akanksha ,

    Welcome to the Microsoft Q&A platform.

    From the error message, java.lang.Exception: Could not find File/xxxxxxxx/clickstream/driverenum/temp", looks like you are passing incorrect path.

    Make sure the path exists and try the steps mentioned below and do let us know the status.

    OR

    You can use this method for access ADLS Gen2 files directly using the RDD API as shown below.

    15851-image.png

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.