Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,514 questions
Creating Spark database in Azure Databricks with location to ADLS Gen 2 using ABFS driver throws an exception
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 20%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Ramesh
50
Reputation points
Hi Team, I am creating a database in Azure Databricks using the abfss location in the create table statement and it throws an exception.
- Authentication to ADLS - Session Scoped Access Key Authentication as below
- Access Method to ADLS - abfs driver as below
spark.conf.set("fs.azure.account.key.formula1supportdl.dfs.core.windows.net",
dbutils.secrets.get(scope="databricks-support-scope", key="formula1supportdl-account-key"))
%sql
CREATE DATABASE f1_demo_abfss
LOCATION 'abfss://******@formula1supportdl.dfs.core.windows.net/'
Error message as below
Additional information
- Access to the same storage account/ container works perfectly fine if I use the cluster scoped authentication. i.e., when adding the spark configuration to the cluster configuration instead of the using in the notebook.
- Specifying the location for a table as above works perfectly fine too. This is only for a problem in create database statement.
Example CREATE TABLE with the same location works perfectly fine as below
As this is only a problem when trying to access the location on CREATE DATABASE statement using session scoped authentication, it seem to be a bug. As I said about cluster scoped authentication works perfectly fine. Also, CREATE TABLE with the same location works perfectly fine.
Can this be looked into please? Let me know if you need more information to investigate.
Thanks
Azure Databricks
{count} votes
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator2023-02-14T09:40:50.61+00:00 Hello @Ramesh
Thanks for the question and using MS Q&A platform.
Could you please share the stack trace of the error message which you are experiencing?
-
Ramesh 50 Reputation points
2023-02-15T15:47:42.9566667+00:00 Hi Pradeep,
Thanks for looking into this issue.
Please see the information requested as below
--------------------------------------------------------------------------- AnalysisException Traceback (most recent call last) <command-2302823602205408> in <cell line: 1>() 5 display(df) 6 return df ----> 7 _sqldf = ____databricks_percent_sql() 8 finally: 9 del ____databricks_percent_sql <command-2302823602205408> in ____databricks_percent_sql() 2 def ____databricks_percent_sql(): 3 import base64 ----> 4 df = spark.sql(base64.standard_b64decode("Q1JFQVRFIERBVEFCQVNFIGYxX2RlbW9fYWJmc3MKTE9DQVRJT04gJ2FiZnNzOi8vZGVtb0Bmb3JtdWxhMXN1cHBvcnRkbC5kZnMuY29yZS53aW5kb3dzLm5ldC8n").decode()) 5 display(df) 6 return df /databricks/spark/python/pyspark/instrumentation_utils.py in wrapper(*args, **kwargs) 46 start = time.perf_counter() 47 try: ---> 48 res = func(*args, **kwargs) 49 logger.log_success( 50 module_name, class_name, function_name, time.perf_counter() - start, signature /databricks/spark/python/pyspark/sql/session.py in sql(self, sqlQuery, **kwargs) 1117 sqlQuery = formatter.format(sqlQuery, **kwargs) 1118 try: -> 1119 return DataFrame(self._jsparkSession.sql(sqlQuery), self) 1120 finally: 1121 if len(kwargs) > 0: /databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in __call__(self, *args) 1319 1320 answer = self.gateway_client.send_command(command) -> 1321 return_value = get_return_value( 1322 answer, self.gateway_client, self.target_id, self.name) 1323 /databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw) 200 # Hide where the exception came from that shows a non-Pythonic 201 # JVM exception message. --> 202 raise converted from None 203 else: 204 raise AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException Failure to initialize configuration) -
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2023-02-16T08:23:00.9133333+00:00 Hello @Ramesh
This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.
-
Ramesh 50 Reputation points
2023-02-16T11:34:19.74+00:00 Thanks for the response. Unfortunately, I do not have a support plan. So, I will wait for your response here.
Do you have a view on how long it's likely to take for a response via this route?
-
Ramesh 50 Reputation points
2023-02-23T17:05:23.9833333+00:00 -
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2023-02-24T05:00:28.8566667+00:00 Hello @Ramesh
We didn't get any response from the internal team, I will update you post receiving an update.
-
Ramesh 50 Reputation points
2023-04-13T08:55:54.25+00:00 Hi @PRADEEPCHEEKATLA - Just checking to see if you have any update from your team? Thanks
Sign in to comment
Sign in to answer