com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token

Question

I am using databricks in Azure, we have a cluster in it with credentials passthru enabled prior and we have mounted mount points using below code:

configs = {
  "fs.azure.account.auth.type": "CustomAccessToken",
  "fs.azure.account.custom.token.provider.class": spark.conf.get("spark.databricks.passthrough.adls.gen2.tokenProviderClassName")
}

dbutils.fs.mount(
  source = "abfss://******@az21p1datalakewe.dfs.core.windows.net/gold",
  mount_point = "/mnt/testdev/",
  extra_configs = configs)

But few days ago , someone accidentaly deleted all spark-configs in the same clutser. After we are facing this below error, when accessing adls via mount points.

com.databricks.backend.daemon.data.common.InvalidMountException: Error while using path.....

Then we added these below 2 configs in cluster and tried to create mount points again using credentials pass thru method which throws below error.

com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token

Below are the spark-configs we added in the cluster now, not sure what it had before when it was working fine.

spark.databricks.passthrough.enabled true

spark.databricks.repl.allowedLanguages python,sql

Could you please help on this whether we missing any extra config needed to add in cluster. Thanks!

Answer 1

Hello Vigneshwaran Kumaravel, I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others", I'll repost your solution in case you'd like to "Accept" the answer.

Issue:

Missing spark-configs Databricks cluster has credentials passthru enabled and mounted mount points using below code:

configs = {
  "fs.azure.account.auth.type": "CustomAccessToken",
  "fs.azure.account.custom.token.provider.class": spark.conf.get("spark.databricks.passthrough.adls.gen2.tokenProviderClassName")
}

dbutils.fs.mount(
  source = "abfss://******@az21p1datalakewe.dfs.core.windows.net/gold",
  mount_point = "/mnt/testdev/",
  extra_configs = configs)

Unable to access mount points after missing spark-configs.

Error: com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token

Solution:

The issue was resolved by adding the below spark configs.

spark.databricks.passthrough.enabled true spark.databricks.pyspark.enableProcessIsolation true spark.databricks.cluster.profile serverless spark.databricks.repl.allowedLanguages python,sql

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.