Creating database with location ADLS Gen 2 using abfss generates exception

Question

Hello,

I am creating a database in Azure Databricks using the abfss location in the create table statement and it throws an exception.

%sql

CREATE DATABASE IF NOT EXISTS test COMMENT "Database for Test Area" LOCATION "abfss://******@storagetemp.dfs.core.windows.net/database/"

I am using this method, I created also the Scope:

Azure Service Principal

service_credential = dbutils.secrets.get(scope="Test",key="client_secret")

client_id = dbutils.secrets.get(scope="Test", key="client_id")

tenant_id = dbutils.secrets.get(scope="Test", key="tenant_id")

spark.conf.set("fs.azure.account.auth.type.storagetemp.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.storagetemp.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id.storagetemp.dfs.core.windows.net", f"{client_id}")

spark.conf.set("fs.azure.account.oauth2.client.secret.storagetemp.dfs.core.windows.net", f"{service_credential}")

spark.conf.set("fs.azure.account.oauth2.client.endpoint.storagetemp.dfs.core.windows.net", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")

Everthing is well when I am trying to do a .save(), also:

location = "abfss://******@storagetemp.dfs.core.windows.net/"

dbutils.fs.ls(location)

Out[27]: [FileInfo(path='abfss://******@storagetemp.dfs.core.windows.net/database/', name='database/', size=0, modificationTime=1691162354000)]

works fine.

The error from creating the database is:

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException Failure to initialize configuration for storage account storagetemp.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key)

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException Failure to initialize configuration for storage account storagetemp.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key)
---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
File 

Answer 1

@Mihai Cosmin - Thanks for the question and using MS Q&A paltform.

Before we get into resolving the issue - please do confirm the below information:

Are you using Databricks compute or SQL datawarehouse to run the above commands?

As metioned in the question, you are using Azure Service principal but as per the error message Invalid configuration value detected for fs.azure.account.keyplease confirm which method is used to configure the storage account - Azure Service principal or Storage account keys?

Coming to the resolving the issue:

It looks like you are encountering an error while trying to initialize the configuration for a storage account in Azure Databricks. The error message suggests that there is an invalid configuration value detected for the fs.azure.account.key parameter.

To resolve this issue, you can try the following steps:

1. Check if the storage account name and key are correct. Make sure that there are no typos or errors in the values.
2. Verify that the storage account key is valid and has not expired. You can regenerate the key if necessary.
3. Check if the storage account is accessible from Azure Databricks. You can try accessing the storage account using the Azure Storage Explorer or another tool to confirm that it is accessible.
4. Ensure that the storage account is correctly configured for use with Azure Databricks. You can refer to the Azure Databricks documentation for more information on how to configure storage accounts.

If none of the above steps resolve the issue, you may open a support for further assistance.

As per the repro from my end, i'm able to create database with location ADLS Gen 2 using abfss as shown below:

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.