Mount Points in Databricks with Key Vault secret rotation

Question

I created a secret in Key Vault storing the credentials for an ADLS. I need to get access to a blob in the ADLS from Databricks. The problem is that every month, the keys are rotated for security reasons. If I understood it correctly, if I create a secret scope in Databricks and the keys are rotated after one month, they are not refreshed/updated automatically in the secret scope? Is there a workaround or a different method to get access from Databricks to the ADLS? If I use the Databricks CLI with this command:

databricks secrets create-scope --scope <scope-name> --scope-backend-type AZURE_KEYVAULT --resource-id <azure-keyvault-resource-id> --dns-name <azure-keyvault-dns-name> --initial-manage-principal users

it won't work either correct? I was wondering because here the resource ID and Vault URI are used. Thank you in advance!

Answer 1

Wentzler, Charlotte - Thanks for the question and using MS Q&A platform.

Yes, you are correct that if you create a secret scope in Databricks and the keys are rotated after one month, they are not automatically updated in the secret scope. To work around this issue, you can use the Key Vault-backed secret scope feature in Databricks.

With Key Vault-backed secret scope, the secrets are fetched from the Key Vault at runtime, which means that the secrets are always up-to-date with the latest version in the Key Vault. This feature provides automatic rotation of secrets in the Key Vault and ensures that the secrets used by your Databricks workloads are always up-to-date.

For more details, refer to the below MS Q&A thread addressing similar issue:

https://learn.microsoft.com/en-us/answers/questions/1185280/databricks-secretscope-secret-versions

Hope this helps. Do let us know if you any further queries.