Hadoop Azure Support: Gen2

Question

Can we connect to gen2 storage via sas token like we do azure storage blob?

Generally, we do hadoop configuration azure storage blob via sas token as :

sc.hadoopConfiguration.set("fs.azure.sas." + accountName + ".blob.core.windows.net", sasToken)

sc: spark session.

Do we have something similar for gen2 storage? I know we have OAuth 2.0 credential support and account key, but I am looking especially for sas token support for gen2 connector.

Answer 1

Hello @Tansheet Izhad ,

Welcome to the Microsoft Q&A platform.

Does ABFS support Shared Access Signature (SAS) token authentication?

ABFS does not support SAS token authentication.

Note: You can access an Azure Data Lake Storage Gen2 storage account using the storage account access key.

spark.conf.set("fs.azure.account.key.<storage-account-name>.dfs.core.windows.net", "Storage_Account_Access_Key")  

This article helps you to DIY: Apache Spark and ADLS Gen 2 support, make sure you have followed all the necessary steps to successfully configure ADLS gen2 on Hadoop - Apache Spark and Accessing Azure Data Lake Storage Gen2 From Clients.

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.