Creating database with location to abfss generates exception

Question

I am trying to create a database in Azure Databricks where I specify the location parameter to ADLS G2 but get exception

This is the spark conf which I execute first:

spark.conf.set("fs.azure.account.auth.type.<storage-account-name>.dfs.core.windows.net", "SAS")  
spark.conf.set("fs.azure.sas.token.provider.type.<storage-account-name>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")  
spark.conf.set("fs.azure.sas.fixed.token.<storage-account-name>.dfs.core.windows.net", dbutils.secrets.get(scope="scope", key="key"))  

Note: to access Azure storage, I am using SAS. I have not mounted the storage.

The following is the database creation statement:

CREATE DATABASE IF NOT EXISTS processed  
LOCATION "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"  

But it generates the following exception:

Error in SQL statement: AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException Failure to initialize configuration)  
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException Failure to initialize configuration)  
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$2(HiveExternalCatalog.scala:163)  
	at org.apache.spark.sql.hive.HiveExternalCatalog.maybeSynchronized(HiveExternalCatalog.scala:115)  
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$1(HiveExternalCatalog.scala:153)  
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:377)  
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:363)  
	at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34)  
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:152)  
	at org.apache.spark.sql.hive.HiveExternalCatalog.createDatabase(HiveExternalCatalog.scala:271)

Answer 1

Hello @Aqa Mustafa Akhlaqi ,

Welcome to the MS Q&A platform.

You will need to provide configuration parameters for ADLS authentication in the SQL Admin console, as described in the documentation below.

https://learn.microsoft.com/en-us/azure/databricks/sql/admin/data-access-configuration#--configure-a-service-principal

I hope this helps.

------------------------------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators