How to access Azure Devops from Databricks using a Service Principal

Question

How to access Azure Devops from Databricks using a Service Principal

Lilac 80

Hi. I am trying to install a package (Flask==2.0.2) from my Azure Devops portal using Databricks. For this purpose I am using a PAT and passing this in the %pip install statement in databricks. Now I have created a Service principal and used the same in my Service Connections in Azure Devops, but while using this, I am not able to run my pip install. I have scoured the net for a possible solution. Can you please help.

Konstantinos Passadis 19,701 Reputation points MVP

2023-07-09T21:26:40.0566667+00:00

Hello @Kunal Chowdhury !

Do you have any feedback or is your issue resolved?

In case any answer helped kindly mark it as Accepted and upvote so others can benefit as well!

Otherwise post your feedback to provide additional help!

Regards

Answer accepted by question author

0 additional answers

Your answer

Konstantinos Passadis 19,701 Reputation points MVP

2023-07-09T21:26:40.0566667+00:00

Hello @Kunal Chowdhury !

Do you have any feedback or is your issue resolved?

In case any answer helped kindly mark it as Accepted and upvote so others can benefit as well!

Otherwise post your feedback to provide additional help!

Regards

Answer 1

Konstantinos Passadis 19,701 MVP

Hello @Kunal Chowdhury !

Welcome to Microsoft QnA!

Please tell us does the Service Principal has the Contributor Role ?

Also there is another thing you can try :

In Azure AD , go to Application Registrations , find your Service Principal and click API Permissions

Click ADD PERMISSIONS and select the DevOps API and add user_impersonation

Grant AdminConsent and try again , although we need more details on th error you are getting

what are the errors you are encountering ?

The more details we have the better help we can provide !

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Lilac 80 Reputation points

2023-07-10T06:07:55.38+00:00

Hi @Konstantinos Passadis . Thank you so much for your response. Let me give you an elaborate description of the problem. I am trying to install a package from my Devops Repo in a databricks notebook. While using a PAT token generated from Devops, the following statement was able to download and install it.

%pip install --index-url https://$user:$******@pkgs.dev.azure.com/SmartFactory_Feed/pypi/simple Flask==2.0.2 --no-cache-dir

Please note that $user and $token were stored as databricks scoped credentials and are the PAT username and token.

However, to implement Microsoft best practices, I am planning to relace the PAT approach with a Service Principal. While I created the SP and gave it controbuter access in Azure Devops Service Connections, I was not able to run this statement.

I simply replaced the $user:$token with the SP's client id and client secret.

Can you please tell me a better way to do this. I have not tried out the above approach that you suggested as I am stil waiting on my IT team to give the Devops API permission.
Konstantinos Passadis 19,701 Reputation points MVP

2023-07-12T18:27:48.74+00:00
Hello @Kunal Chowdhury !

Did you get the permissions ? Reading your issue it points me there , to the procedure i posted

BUT

I have found this :

Use a user interface

To create a Databricks service principal by using the Databricks user interface, see Add a service principal to your Databricks account and Add a service principal to a workspace.

To create a Databricks access token for a Databricks service principal, see Manage personal access tokens for a service principal. You cannot use the Databricks user interface for this step.

ssign a service principal to a workspace using the account console

As an account admin, log in to the account console. In the sidebar, click Workspaces. Click your workspace name. On the Permissions tab, click Add permissions. Search for and select the service principal, assign the permission level (workspace User or Admin), and click Save.

Add a service principal to a workspace using the workspace admin settings

To add a service principal to a workspace using the workspace admin settings page, the workspace must be enabled for identity federation.

As a workspace admin, log in to the Databricks workspace. Click your username in the top bar of the Databricks workspace and select Admin Settings. On the Service principals tab, click Add service principal. Select an existing service principal to assign to the workspace or create a new one. To create a new service principal, click the drop-down arrow in the search box and then click + Add new service principal.

Note

If your workspace is not enabled for identity federation, you cannot assign existing account service principals to your workspace.

Assign a service principal to a workspace using REST APIs

The REST APIs that you can use to assign service principals to workspaces depend on whether the workspace is enabled for identity federation:

Workspace enabled for identity federation: Account and workspace admins can use the Workspace Assignment API to assign service principals to workspaces. See Workspace Assignment API. Workspace not enabled for identity federation: A workspace admin can use the workspace-level SCIM (ServicePrincipal) API to assign service principals to their workspaces. See the Service Principal (SCIM) API.

Assign workspace admin rights to a service principal

You can assign the workspace admin role using the account console, workspace admin settings page, or REST APIs.

Assign the workspace admin role to a service principal using the account console

To assign the workspace admin role using the account console, the workspace must be enabled for identity federation.

As an account admin, log in to the account console. Click Workspace Icon Workspaces. On the Permissions tab, find the service principal. Click the Kebab menu kebab menu at the far right of the user row and select Edit. Under Role, choose Admin. Click Save.

To remove the admin role from a workspace service principal, perform the same steps, but choose User under Role.

Assign the workspace admin role to a service principal using the workspace admin settings page

To assign the workspace admin role using the workspace admin settings page, do the following:

As a workspace admin, log in to the Databricks workspace. Click your username in the top bar of the Databricks workspace and select Admin Settings. On the Groups tab, select the Admins group. Click Add users or service principals. Select the service principal and click Confirm.

To remove the workspace admin role from a service principal, remove the service principal from the admin group.

Assign the workspace admin role to a service principal using the REST APIs

The REST APIs that you can use to assign the workspace admin role depend on whether the workspace is enabled for identity federation:

Workspace enabled for identity federation: An account admin can use the account-level Workspace Assignment API to assign or remove the workspace admin role. Either an account admin or workspace admin can use the workspace-level Workspace Assignment API to perform this task. Workspace not enabled for identity federation: A workspace admin can use the workspace-level SCIM (Groups) REST API to assign a service principal to the admin group or remove them from the group. See the Service Principals API.

LINK: https://docs.databricks.com/administration-guide/users-groups/service-principals.html#add-sp-workspace

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards
Lilac 80 Reputation points

2023-07-17T07:27:13.1633333+00:00

Thanks @Konstantinos Passadis This really helps!!

Share via

How to access Azure Devops from Databricks using a Service Principal

0 additional answers

Your answer