How to access files from blob storage using SAS token in databricks ?

Prasanna, Prashanth (RIS-CON) 20 Reputation points
2023-05-10T14:37:41.3766667+00:00

I have tried the below code. It works fine using wasbs. but I am getting error when I tried to read it using abfss

storage_account_name = "newcontainer"

container_name = "source"

sas_token = "mysastoken"

data_path = f"wasbs://{container_name}@{storage_account_name}.blob.core.windows.net/Test/file.csv"

src_path = f"abfss://{container_name}@{storage_account_name}.blob.core.windows.net/Test/file.csv"

Set the necessary configuration for Spark

spark.conf.set(

    f"fs.azure.sas.{container_name}.{storage_account_name}.blob.core.windows.net",

    sas_token

)

Read data from the Blob container

df = spark.read.csv(data_path, header=True, inferSchema=True)

The above code works fine but If I use abfss ,

df = spark.read.csv(src_path , header=True, inferSchema=True)

I am getting the error: "This request is not authorized to perform this operation using this permission.", 403

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,561 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,030 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 83,301 Reputation points Microsoft Employee
    2023-05-11T08:45:59.0966667+00:00
    • Thanks for the question and using MS Q&A platform.

    [UPDATE]: Yes, you are correct. As per the repro - you will experience -
    Failure to initialize configuration for storage account cheprademo.blob.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key.

    User's image

    This is a known issue that - you cannot use storage account SAS token to access data using the abfss protocol.

    Reason: You can't use storage account SAS Token to access data using the abfss protocol. You need to provide more configuration options if you want to use abfss - it's all described in documentation.

    For more details, refer to the below links:

    https://stackoverflow.com/questions/69825487/error-invalid-configuration-value-detected-for-fs-azure-account-key
    https://medium.com/@kyle.hale/troubleshooting-invalid-configuration-value-detected-for-fs-azure-account-key-6c6fcc67b217

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.


    As per the repro from our end, it's working as expected without any issue as shown below.

    Tested on: Databricks Runtime Version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)

    User's image

    In order to investigate further, kindly share the below details.

    Databricks Runtime version?

    Complete stack trace of the error message which you are experiencing?

    For more details, refer to Connect to Azure Data Lake Storage Gen2 and Blob Storage - SAS Token.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.