databricks-adls connection issue, error [CONFIG_NOT_AVAILABLE] Configuration fs.azure.account.auth.type.******tsum*****.dfs.core.windows.net is not available. SQLSTATE: 42K0I

Question

i have trouble connection to data storage (adls) from databricks

i am using below on databricks notebook to connect to adls but every time it gives me error (error [CONFIG_NOT_AVAILABLE] Configuration fs.azure.account.auth.type.*tsum.dfs.core.windows.net is not available. SQLSTATE: 42K0I)

---

service_credential = dbutils.secrets.get(scope="<scope>",key="<service-credential-key>")

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")

spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", <Secret-Value>)

spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

---

i have deleted the existing cluster and created it again, created new application_id, directory_id, secret value, but didn't worked

i ran below on notebook, that worked

storage_account = "******tsum*****"
print(f"{storage_account}.dfs.core.windows.net")

Answer 1

hi,

the error usually means databricks can't find the configuration for your storage account. the most common culprit is a tiny typo in the storage account name within those spark config lines. double check that every instance of <storage-account> matches your actual storage account name exactly. even a single extra character or a missing hyphen will break it.

since your print statement worked, we know the storage account name itself is correct. the issue is likely in how the config is being applied. make sure you're setting these configurations before you try to read any data. a good practice is to put all these spark.conf.set lines in a single cell at the very top of your notebook and run it first.

also, check your secret scope. ensure that the scope name and key in dbutils.secrets.get are exactly right. if the secret isn't fetched correctly, the client secret would be wrong, causing the auth to fail.

this might help in other tools too, always double check your config keys for typos.

if everything looks correct, try using the abfss path directly in a read operation to see if the error changes. sometimes a different error message can give a better clue.

df = spark.read.format("csv").load("abfss://<container>@<storage-account>.dfs.core.windows.net/<path>")

aha, and one more thing. ensure your service principal has the correct permissions on the storage account. it needs at least 'storage blob data contributor' role on the specific container you're trying to access.

hope this connects the dots for you. those config errors are always so subtle. let me know if you spot a typo.

Best regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/