Azure Databricks mounts using Azure KeyVault-backed scope -- SP secret update

Question

Issue:

1. Mounted ADLS gen2 container using service principal secret as secret from Azure Key Vault-backed secret scope. All good, can access the data.
2. Deleted secret from service principal in AAD, added new, updated Azure Key Vault secret (added the new version, disabled the old secret). All was still good, could access the data.
3. Restarted cluster. Unable to access mount point, error: “AADToken: HTTP connection failed for getting token from AzureAD. Http response: 401 Unauthorized”
4. Unmount/mount using the same config helped.

Is there a way to refresh the secret used for mount point that I could add to init scripts to avoid this issue? I would rather avoid unmounting/mounting all mount points in init scripts and was hoping that there is something like dbutils.fs.refreshMounts() (refreshMounts didn't help with this) that would help.

I mounted ADLS Gen2 using service principal, oauth2.0, and azure key vault-backed secret scope, following this documentation: https://learn.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-datalake-gen2#mount-azure-data-lake-gen2

I was also wondering - how long does the token live for accessing the ADLS Gen2? I changed the service principal secret (deleted previous one, added new) from AAD and was able to access the mnt point data for more than 12 hours after that (until cluster was restarted).

Answer 1

Hi GuntaLedina-8853, this is a known limitation. Whenever you create a mount point using credentials coming from an AKV backed secret scope, the credentials will be stored in the mount point and will never be refreshed again. This is a one-time read activity on mount point creation time. So each time you rotate credentials in AKV you need to re-create the mount points to refresh the credentials there.

You could add this as a feature request: https://feedback.azure.com/forums/909463-azure-databricks