How to access files from blob storage using SAS token in databricks ?

Question

I have tried the below code. It works fine using wasbs. but I am getting error when I tried to read it using abfss

storage_account_name = "newcontainer"

container_name = "source"

sas_token = "mysastoken"

data_path = f"wasbs://{container_name}@{storage_account_name}.blob.core.windows.net/Test/file.csv"

src_path = f"abfss://{container_name}@{storage_account_name}.blob.core.windows.net/Test/file.csv"

Set the necessary configuration for Spark

spark.conf.set(

    f"fs.azure.sas.{container_name}.{storage_account_name}.blob.core.windows.net",

    sas_token

)

Read data from the Blob container

df = spark.read.csv(data_path, header=True, inferSchema=True)

The above code works fine but If I use abfss ,

df = spark.read.csv(src_path , header=True, inferSchema=True)

I am getting the error: "This request is not authorized to perform this operation using this permission.", 403

Answer 1

• Thanks for the question and using MS Q&A platform.

[UPDATE]: Yes, you are correct. As per the repro - you will experience -
Failure to initialize configuration for storage account cheprademo.blob.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key.

This is a known issue that - you cannot use storage account SAS token to access data using the abfss protocol.

Reason: You can't use storage account SAS Token to access data using the abfss protocol. You need to provide more configuration options if you want to use abfss - it's all described in documentation.

For more details, refer to the below links:

https://stackoverflow.com/questions/69825487/error-invalid-configuration-value-detected-for-fs-azure-account-key
https://medium.com/@kyle.hale/troubleshooting-invalid-configuration-value-detected-for-fs-azure-account-key-6c6fcc67b217

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

---

As per the repro from our end, it's working as expected without any issue as shown below.

Tested on: Databricks Runtime Version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)

In order to investigate further, kindly share the below details.

Databricks Runtime version?

Complete stack trace of the error message which you are experiencing?

For more details, refer to Connect to Azure Data Lake Storage Gen2 and Blob Storage - SAS Token.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.