AD Authentication to Databricks REST API Returns 403 Unauthorized

Question

I'm attempting to call the Databricks API using the method described in Get an Azure Active Directory token using a service principal.

However when I use the management endpoint access token to access the Databricks REST API, I get HTTP ERROR 403. User Not Authorized.

The first 2 calls successfully return w/the appropriate access tokens. These are passed into the Databricks REST API per the documentation, however it returns 403. Help please.

# Attempting this   
#   * https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token  
  
tenantId="secret"  
subscriptionId="secret"  
resourceGroupId="my-test-group"  
clientId="secret"  
clientSecret="verysecret"  
dbResourceId="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"  
dbWorkspaceName="my-test-db"  
  
az login -u $1 -o table  
  
  
dbAccessToken=$(curl -X GET \  
-H 'Content-Type: application/x-www-form-urlencoded' -d  \  
'grant_type=client_credentials&client_id='${clientId}'&resource='${dbResourceId}'&client_secret='${clientSecret} \  
https://login.microsoftonline.com/${tenantId}/oauth2/token | jq -r '.access_token')  
  
echo Access Token  
echo $dbAccessToken  
mgtAccessToken=$(curl -X GET \  
-H 'Content-Type: application/x-www-form-urlencoded' -d \  
'grant_type=client_credentials&client_id='${clientId}'&resource=https://management.core.windows.net/&client_secret='${clientSecret} \  
https://login.microsoftonline.com/${tenantId}/oauth2/token | jq -r '.access_token')  
  
  
echo   
echo Management Access Token  
echo $mgtAccessToken  
echo  
result=$(curl -X GET \  
-H 'Authorization: Bearer '${dbAccessToken} \  
-H 'X-Databricks-Azure-SP-Management-Token: '${mgtAccessToken} \  
-H 'X-Databricks-Azure-Workspace-Resource-Id: /subscriptions/'${subscriptionId}'/resourceGroups/'${resourceGroupId}'/providers/Microsoft.Databricks/workspaces/'${dbWorkspaceName} \  
https://"secretworkspaceid".azuredatabricks.net/api/2.0/clusters/list)  
  
echo  
echo $result  

Answer 1

@M.Lo check whether you have a Contributor or Owner role assigned to your service principal

az role assignment create --assignee $(clientId) --role "Contributor" --scope subscriptions/$(subscriptionId)/resourceGroups/$(resourceGroupName)/$(resourceName)  

Answer 2

I have similar issue but when using token generated with https://....azuredatabricks.net/api/2.0/token/create. My service principal have a Contributor role assigned, then I'm creating a token and saving this token to key vault. I'd like to consume this token in Azure Data Factory but then I'm getting 403 when testing connection via Databricks linked service. On the other hand when I create personal access token via Databricks portal and provide it to ADF then it works.

The main issue is that we have this generation of access token automated via pipeline and manual way of getting it isn't an option for us.

Any ideas?