Mount ADLS Gen2 storage to DBFS with SAS token

Question

Mount ADLS Gen2 storage to DBFS with SAS token

P, John 240

According to the link https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage, user can access the ADLS Gen2 with OAuth2, SAS token, and Account key. To mount the ADLS to DBFS, the document https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts, listed on OAuth2. Is it possible to mount ADLS Gen2 to DBFS with SAS or Account key? If so, how to do it?

Accepted answer

0 additional answers

Your answer

Answer 1

AnnuKumari-MSFT 34,556 Microsoft Employee Moderator

Hi Pan, John , Thankyou for using Microsoft Q&A platform and thanks for posting your query here. As per my understanding you are looking for way to mount ADLS gen2 using Account Key or SAS token in Azure Databricks. This can be achieved using the following code snippet : Account Key:

   dbutils.fs.mount(
      source = "wasbs://{containerName}@{storageAccountName}.blob.core.windows.net",
      mount_point = mountPoint,
      #extra_configs = {'fs.azure.account.key.' + storageAccountName + '.blob.core.windows.net': storageAccountAccessKey}

User's image

SAS token:

   dbutils.fs.mount(
      source ="wasbs://
      mount_point = mountPoint,
      extra_configs = {'fs.azure.sas.' + blobContainerName + '.' + storageAccountName + '.blob.core.windows.net': sasToken}

User's image

For more details, kindly watch out the below resources: Create Mount point using dbutils.fs.mount() in Azure Databricks Mount Azure Blob Storage to DBFS in Azure Databricks

Without having to create mount point, you can set Spark properties to configure Azure credentials to access Azure storage both using access key and SAS token , watch out the below videos for implementation : Access ADLS Gen2 storage using Account Key in Azure Databricks Access ADLS Gen2 or Blob Storage using a SAS token in Azure Databricks Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

P, John 240 Reputation points

2024-02-12T19:55:53.33+00:00

Thanks for the quick response! To my understanding, wasbs has been deprecated, user shall move to abfss instead. Will the SAS token or Access key works for abfss driver?
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-02-13T06:31:44.6766667+00:00
Hi Pan, John , It seems mounting usingabfss might require OAuth authentication , when I tried from my end using account key is throwing an error. Here is the code snippet for using abfss:

configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", "fs.azure.account.oauth2.client.id": "<application-id>", "fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="<scope-name>",key="<service-credential-key-name>"), "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<directory-id>/oauth2/token"} # Optionally, you can add <directory-name> to the source URI of your mount point. dbutils.fs.mount( source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/", mount_point = "/mnt/<mount-name>", extra_configs = configs)

You can either go for this approach or else directly set Spark properties to configure Azure credentials to access Azure storage both using access key and SAS token as suggested above . Watch out the below videos for implementation : Access ADLS Gen2 storage using Account Key in Azure Databricks Access ADLS Gen2 or Blob Storage using a SAS token in Azure Databricks Hope it helps . Kindly accept the answer by clicking on Accept answer button. Thankyou

P, John 240

I figured it out, mount the abfss with SAS is working for me now (see example code below). I haven't tried to mount abfss with Access key ( not a safe method for us). At least we know both OAuth and SAS mountings work for abfss.

sasToken="***"
dbutils.fs.mount(source = "abfss://<container-name>@<storage-account>.dfs.core.windows.net", mount_point = "/mnt/landing",extra_configs = {'fs.azure.account.auth.type':'SAS', 'fs.azure.sas.token.provider.type': 'org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider','fs.azure.sas.fixed.token': sasToken})
dbutils.fs.ls("/mnt/landing")

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-02-19T10:49:20.4633333+00:00

Hi Pan, John , Glad to know that your issue has been resolved. Kindly take a moment to accept the answer by clicking on Accept answer button, in case you found it helpful. Thankyou

Share via

Mount ADLS Gen2 storage to DBFS with SAS token

0 additional answers

Your answer