Azure Databricks mounts using Azure KeyVault-backed scope -- SP secret update

Gunta Ledina 26 Reputation points


  1. Mounted ADLS gen2 container using service principal secret as secret from Azure Key Vault-backed secret scope. All good, can access the data.
  2. Deleted secret from service principal in AAD, added new, updated Azure Key Vault secret (added the new version, disabled the old secret). All was still good, could access the data.
  3. Restarted cluster. Unable to access mount point, error: “AADToken: HTTP connection failed for getting token from AzureAD. Http response: 401 Unauthorized”
  4. Unmount/mount using the same config helped.

Is there a way to refresh the secret used for mount point that I could add to init scripts to avoid this issue? I would rather avoid unmounting/mounting all mount points in init scripts and was hoping that there is something like dbutils.fs.refreshMounts() (refreshMounts didn't help with this) that would help.

I mounted ADLS Gen2 using service principal, oauth2.0, and azure key vault-backed secret scope, following this documentation:

I was also wondering - how long does the token live for accessing the ADLS Gen2? I changed the service principal secret (deleted previous one, added new) from AAD and was able to access the mnt point data for more than 12 hours after that (until cluster was restarted).

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,386 questions
{count} votes

Accepted answer
  1. Jorg Klein 91 Reputation points

    Hi GuntaLedina-8853, this is a known limitation. Whenever you create a mount point using credentials coming from an AKV backed secret scope, the credentials will be stored in the mount point and will never be refreshed again. This is a one-time read activity on mount point creation time. So each time you rotate credentials in AKV you need to re-create the mount points to refresh the credentials there.

    You could add this as a feature request:

    3 people found this answer helpful.

0 additional answers

Sort by: Most helpful