Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,514 questions
Databricks credential passthrough error: com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 96%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELA%3C/text%3E%3C/svg%3E)
Leyi Au Yeung
6
Reputation points
We are running Databricks jobs on single-node clusters with credential passthrough. The Databricks runtime version is: 10.2 ML (includes Apache Spark 3.2.0, Scala 2.12). After a couple of days the job will fail with the following error message:
com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token
Does someone know why this happens and how to prevent this?
Azure Databricks
{count} vote
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator2022-04-26T05:23:04.193+00:00 Hello @Leyi Au Yeung ,
Thanks for the question and using MS Q&A platform.
Could you please share the region of your databricks workspace?
Kindly share more details on steps that lead to the current issue and also a screenshot the issue will be help further navigate the issue.
Looking forward to hearing from you.
-
Leyi Au Yeung 6 Reputation points
2022-04-26T07:54:07.853+00:00 Hi @PRADEEPCHEEKATLA ,
We're running the Databricks Workspace in EU West region and it occurs in the following job:
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2022-04-27T10:52:33.603+00:00 Hello @Leyi Au Yeung ,
Could you please share the code snippet which you are authenticating ADLS Gen2 account and also share the complete stack trace of the error message?
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 31%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Joosten, Bart 6 Reputation points2022-04-28T07:52:34.317+00:00 Hello @PRADEEPCHEEKATLA , I am a colleague of Leyi and I am one of the people affected by this problem. I am not sure if this is relevant, but we use credential passthrough in our databricks cluster setup which should take care of any azure storage authentication step right?
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2022-04-28T07:56:26.353+00:00 Hello @Joosten, Bart ,
Yes, you are correct. After configuring Azure Data Lake Storage credential passthrough and creating storage containers, you can access data directly in Azure Data Lake Storage Gen1 using an
adl://path and Azure Data Lake Storage Gen2 using anabfss://path.I'm request you to share the code snippet which you are trying to call from the storage account and also, share the complete stack trace of the error message.
-
Joosten, Bart 6 Reputation points
2022-04-28T08:18:40.91+00:00 This is a python code snippet for example that generates the error. One important note, the error only occurs when the notebooks are initiated either manually or scheduled from a job. If I run cells directly from a notebook I do not get the errors.
port_visits = spark.sql("select * from silver.dtis_port_visits")Here is the stack trace from stderr as a gist 65a7545405997f7bf6c46f9abec1fe5e since the trace is too large for a comment.
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2022-05-02T05:41:56.703+00:00 Hello @Joosten, Bart ,
You can create a text file with the copy the complete stack trace error message and upload here.
-
Joosten, Bart 6 Reputation points
2022-05-02T08:36:26.577+00:00 Couldn't you just have a look at the trace on the gist? GitHub is a microsoft service after all. I feel like this is taking very long before we get a serious response. Anyway I cannot upload the text file, the upload progress bar stops at 0%.
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2022-05-02T08:49:01.1+00:00 Hello @Joosten, Bart ,
I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation.
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2022-05-10T06:27:03.253+00:00 Hello @Joosten, Bart ,
Did you get a chance to open a support ticket on the issue?
-
Joosten, Bart 6 Reputation points
2022-05-11T08:19:28.797+00:00 No I haven't. I don't know how to do this, and I don't manage any of our cloud services. I am just an end user.
Sign in to comment
Sign in to answer