Allow only a specific Azure Databricks instance to connect to keyvault

Question

I want to connect to Azure Keyvault using "Azure Key Vault-backed scopes" in Databricks. Now on the Keyvault side, the way we allow an application to connect is via:

1. RBAC - We are using the RBAC permission model. Add an identity in RBAC and assign a specific role to it.
   AND
2. Firewall - Allow a particular Network or Allow trusted Microsoft services to bypass this firewall

For Datbricks to connect to Keyvault on RBAC we add AzureDatabricks Enterprise Application ID in RBAC but this allows all the Databricks instances deployed in that Tenant. Is there a way we can restrict only a specific Azure Databricsk instance to connect to Keyvault instead of allowing all Databricks?

Eg:
In my Azure subscription, I have the following resources deployed

1. Keyvault(test-akv) - with RBAC permission model and only selected networks are allowed to connect to it.
2. Azure databricks(test-adb) - with Secure cluster connectivity in a managed VNet.

Now to access keyvault secrets from databricks I will :

1. Create an "Azure Key Vault-backed scopes" for that keyvault(test-akv) in Databricks console
2. Once the scope is created. Under keyvault(test-akv)->Networking set "Allow trusted Microsoft services to bypass this firewall" to Yes so that on firewall side Databricks is allowed to connect.
3. Once the connection is allowed we will Add AzureDatabricks ID (Azure AD->Enterprise applications-> Enterprise Application ID) as a "Key Vault Secrets Officer".
4. After connectivity setting create a sample notebook in databricks to access a particular key in keyvualt via secret scope

Now all this works fine but the settings especially on RBAC that we have added in keyvault will allow any databricks instance deployed in my tenant to connect to keyvault. For an Enterprise solution where we have one tenant but multiple subscriptions for each project. this is a security threat that allows all databricks instances to be able to connect t to the keyvault which should not be the case.

So is there a way we can allow a specific instance of databricks to connect to keyvault instead of allowing all?

Answer 1

I agree with jikuja. This does not solve the issue. An enterprise account that has one tenant but multiple subscriptions for different projects can access any keyvault secret by just creating an Azure keyvault backed secret scope.

I tried adding MSI as "Key Vault Secrets Officer" but then I am not able to access keyvault. It gives me a 403 error.

com.databricks.common.client.DatabricksServiceHttpClientException: INVALID_STATE: Databricks could not access keyvault:

Answer 2

Is there an update or solution on the answer OP has? Im having the same issue.

Answer 3

The short answer is: no, you can't but Databricks has extra layer of protection

Scope creation:
When you create key vault-based databricks secret scope the following entry will be added into key vault's access policy list:

{  
  "objectId": "c836dda9-16de-4b63-95ad-42da15a6055a",  
  "permissions": {  
    "secrets": [  
      "get",  
      "list"  
    ]  
  },  
  "tenantId": "902e9b63-xxxx-xxxx-xxxx-3aafbc9e0fb8"  
}  

where c836dda9-16de-4b63-95ad-42da15a6055a is well-known UUID for tenant-wide AzureDatabricks enterprise application.

Effectively the first scope creation per key vault gives all databricks instances on tenant rights to list and read all secrets from the given keyvault.

This is the main issue in this thread. This is in my opinion HUGE problem because documentation does not explain mention anything about scope creation changing key vault properties or access policy list  
 

Scope creation permission checks:
Databricks key vault-backed scope creation will fail if user or identity creating scope does not have proper rights on the given key vault.

Scope creation process requires write access for key vault control layer: Key Vault Contributor, Contributor or Owner role. Also suitable custom roles will be sufficient.

If role to update regular key vault access policy list is not available scope creation will fail:
Unable to grant read/list permission to Databricks service principal to KeyVault: https://xxx.vault.azure.net/

Other mentions::
Scope creation ignores RBAC-based key vault permission management. Regular key vault's access policy addition will be done even if RBAC-based permission management is turned on.

Should we trust databricks scope creation key vault permission check
In my opinion no. Reasoning:

• Usage of tenant-wide enterprise app access policy addition should be documented. Doing this kind of access policy addition silently is not acceptable
• Permission check should be documented. Giving an error message when check/policy addition fails is not proper documentation.

RBAC-based key vault access policy:
If you use RBAC-based key vault access policy you need to add authorizations manually because databricks scope creation supports only regular access policy.
Note that you can give access on secret level if need that.

TLDR:

• No, you cannot give key vault access for only specific databricks accounts
• but databricks scope creation has internal checks that blocks scope creation if creating user has no key vault access

Related issues:

• https://github.com/MicrosoftDocs/azure-docs/issues/97973 - requested documentation update to clarify required permission and internal key vault access check

Answer 4

Any updates on this or even plans to document this issue?