Questions on Databricks Service Principal Login and ADF Pipeline Notebook Execution Identity

Question

Questions on Databricks Service Principal Login and ADF Pipeline Notebook Execution Identity

Saikumar Karri 0

Hi Team,

I have questions regarding Azure Data Factory (ADF) integration to databricks through SP

Running Databricks Notebooks from ADF pipeline

When I trigger a pipeline in ADF that runs a Databricks notebook, the notebook seems to execute under a specific user identity.

How is this user identity determined?
Can we configure/change which user identity the notebook runs as? If yes, how can it be done (for example, by using Service Principal or Managed Identity)?

Thanks in advance for the clarification!

Saikumar Karri 0 Reputation points

2025-09-11T20:18:54.2+00:00

From my understanding, the user who generates the Databricks access token is the one shown as the execution identity in the notebook activity within the ADF pipeline.

please correct me if wrong
Anonymous

2025-09-12T07:35:34.0866667+00:00
Hi @Anonymous

Thank you for contacting to Microsoft QA. and asking these great questions. This explanation gives a clear overview of how Azure Data Factory (ADF) works with Databricks, especially how it handles user identities when running notebooks.

Here’s a detailed response addressing each of your questions: -

·         How is the User Identity Determined?

When you use ADF to run Databricks notebooks, ADF typically integrates with Databricks through a Databricks linked service. There are two common authentication methods to connect ADF to Databricks:

Personal Access Token (PAT) Authentication

In this case, the Databricks Access Token is generated by a user in Databricks (usually through the UI), the identity shown during notebook execution will reflect the user who created that access token. So, your understanding is correct: - the user who generated the token is what Databricks logs as the execution identity.

Service Principal Authentication (Recommended for Production)

A more secure and scalable approach is to configure ADF to use a Service Principal (SP) for authentication to Databricks. In this case, the execution identity will reflect the Service Principal’s identity. The notebook will run under this SP’s context, not a personal user.

·         Can We Configure/Change the Execution Identity?

               Yes, we can control the identity under which the notebook runs, by                configuring ADF to authenticate using a Service Principal or Managed Identity.
Service Principal (SP)

A Service Principal is an identity created in Azure Active Directory for use with applications and automation tools.

You can configure ADF to use a Service Principal by:

·         Creating a Service Principal in Azure AD.

·         Assigning it appropriate roles (e.g., Contributor) in the Databricks workspace.

·         Using the SP credentials (client ID and secret) in the ADF linked service to authenticate to Databricks.

__Managed Identity__

ADF supports both System-Assigned and User-Assigned Managed Identities.

You can assign a Managed Identity to ADF and configure Databricks to recognize it by:

·         Adding the Managed Identity to the Databricks workspace.

·         Granting it necessary permissions (e.g., workspace access, cluster permissions).

·         Using the Access Connector or Unity Catalog (if available) to bridge identity management.

Let me correct you understanding –

When using a Personal Access Token (PAT), the execution identity reflects the user who created the token. However, in the recommended approach using a Service Principal, the notebook runs under the Service Principal’s identity—not a personal user. Always use Service Principal Authentication for predictable, secure execution identity.

I hope this helps clarify your questions regarding the integration of Azure Data Factory (ADF) with Databricks using a Service Principal.

Thanks,
Vrishabh
Saikumar Karri 0 Reputation points

2025-09-12T08:35:40.5366667+00:00

Hi Vrishabh,

Thanks for answering my question. And i have another question related to service principal and databricks,.

Is it possible to login to databricks workspace using service principal?

If yes, can you please let me know how can we do that ?

Thanks ,

Saikumar
Anonymous

2025-09-12T10:15:33.49+00:00
Hi @Saikumar karri

Thanks for reviewing and earlier steps, and here is the answer to your additional question –

Yes, it is possible to use a Service Principal (SP) to authenticate and interact with the Databricks Workspace (especially for automation like running jobs or notebooks).

Prerequisites:

An Azure Active Directory (AAD) Service Principal created in your Azure tenant.

The Service Principal must be assigned proper permissions in Databricks.

Steps to Configure Service Principal Login to Databricks:

Step 1: Create Service Principal in Azure

In Azure Portal → Azure Active Directory → App Registrations → New Registration → Register your App.

Note down the:

Application (Client) ID

Directory (Tenant) ID

Client Secret (from Certificates & Secrets)

Step 2: Assign Service Principal to Databricks Workspace

Go to Databricks Admin Console → Access Control → Service Principals → Add Service Principal.

Enter the SP’s Client ID.

Grant the necessary workspace permissions (e.g., Can Run Jobs, Can Create Clusters, Can Attach Notebooks).

Step 3: Use Service Principal for Authentication

When integrating with Databricks (e.g., in ADF or when using Databricks REST API):

Use the following OAuth 2.0 flow to obtain an Access Token:

POST request to Azure OAuth endpoint: https://login.microsoftonline.com/<Tenant-ID>/oauth2/token

Body includes:
- grant_type = client_credentials - client_id = <Service Principal App ID> - client_secret = <Client Secret> - resource = (Databricks Resource ID) - Use the retrieved Access Token in HTTP Authorization header when making API calls or configuring ADF Linked Service.

Step 4: Configure ADF Databricks Linked Service

Authentication Type → Service Principal

Provide:

Tenant ID

Client ID

Client Secret

Databricks Workspace URL

Official documents for your further reference –

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals

Thanks,
Vrishabh
Saikumar Karri 0 Reputation points

2025-09-12T10:44:35.3366667+00:00

Thanks for answering,

I mean can we open the Databricks UI workspace with service principal like we open with Microsoft entra ID ? And can you please help what all permissions required to get this ?
Anonymous

2025-09-12T11:41:59.8833333+00:00

Hi @Saikumar karri

Service principals in Azure Databricks are API-only identities. They are designed for automation and programmatic access, such as running jobs, accessing data, or managing resources via REST APIs, SDKs, or CLI. They cannot log into the Databricks UI like regular users authenticated via Microsoft Entra ID.

Kindly refer the below official document -

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals

Thanks,
Vrishabh
Saikumar Karri 0 Reputation points

2025-09-12T20:28:32.2633333+00:00

Hi @Vrishabhanath Patil

Thanks for answering, I still not able to find the service principal option in ADF while creating databricks linked service
Anonymous

2025-09-13T15:26:25.8366667+00:00
Hi @Anonymous

Here’s a detailed, step-by-step explanation addressing your questions —

Step-by-Step: Using a Service Principal with ADF and Databricks

·         Create a Service Principal in Azure Entra ID

o   Go to portal.azure.com → Microsoft Entra ID → App registrations → New registration.

o   Provide a name and register the app.

o   Under Certificates & secrets, create a new client secret and save the value securely.

o   Note down:

§ Application (client) ID

§ Directory (tenant) ID

§ Client secret

·         Add the Service Principal to Databricks

o    Go to your Databricks workspace → Settings → Identity and access → Service principals.

o    Click Manage, then Add service principal.

o    Choose Microsoft Entra ID managed, enter the Application ID, and assign necessary permissions (e.g., cluster creation, workspace access).

·         Store Secrets in Azure Key Vault

o    Store the client ID, tenant ID, and client secret in Azure Key Vault.

o    Ensure ADF has access to this Key Vault (via Access Policies or RBAC).

Why “Service Principal” Is Not an Option in ADF Linked Service

ADF’s Databricks linked service does not explicitly list “Service Principal” as an authentication method. Instead, you should use the Access Token option, where the token is generated using the Service Principal credentials.

Workaround:

Use the Service Principal to generate a Personal Access Token (PAT) in Databricks.

Use this PAT in ADF’s linked service under the Access Token authentication method.

This approach is confirmed in both internal documentation and Microsoft Q&A threads.

Alternative (Recommended): Use Managed Identity

If your ADF and Databricks are in the same tenant, Managed Identity is the preferred and more secure method:

Assign ADF’s System Assigned Managed Identity or a User Assigned Identity to the Databricks workspace.

Grant it appropriate roles (e.g., Contributor) and permissions in Databricks.

Then, select Managed Identity or User Assigned Identity in the ADF linked service.

Authentication Options in ADF for Databricks

·         Access Token: Use a PAT generated by a Service Principal (manual token management).

·         Managed Identity: Preferred for secure, seamless integration (requires RBAC setup).

·         User Assigned Identity: Use when multiple ADFs or services share the same identity.

__
Thanks,
Vrishabh__
Saikumar Karri 0 Reputation points

2025-09-13T16:47:47.77+00:00

Hi Vrishabh,

I have tried the provided steps to create access token but

I could not find a way to create a access token through service principal

Can you help me with detailed steps or documentation

Thanks

Saikumar
Anonymous

2025-09-14T13:57:40.1733333+00:00

Hi @Saikumar karri

Thank you for your patience and cooperation. I kindly request you to review the recommended official document provided below thoroughly.

https://learn.microsoft.com/en-us/azure/databricks/security/secrets/

After storing the secrets in Key Vault using the above document, please refer to the screenshot below to use those secrets when creating a linked service. Ensure you select the highlighted 'Authentication type' and provide the required Key Vault parameters.

Thanks,
Vrishabh
Anonymous

2025-09-15T11:50:50.5433333+00:00

Hi @Saikumar karri

Just checking in to confirm whether the steps provided above have resolved your query, or if you need any further assistance from our side.

Thanks,
Vrishabh
Saikumar Karri 0 Reputation points

2025-09-15T19:00:21.61+00:00

Hi @Vrishabhanath Patil ,

I logged in to Databricks using my email account, added the service principal, and assigned it cluster creation and workspace access permissions. After that, I generated a token, stored it in Key Vault, and used it as per the steps mentioned above. However, when I run the notebook, it still shows my user email instead of the service principal.

Thanks,

Saikumar
Anonymous

2025-09-18T10:59:28.29+00:00

Hi @Anonymous

Please confirm, if above given steps are helpful or do you need any further assistance.

Thanks,
Vrishabh

2 answers

Your answer

Saikumar Karri 0 Reputation points

2025-09-11T20:18:54.2+00:00

From my understanding, the user who generates the Databricks access token is the one shown as the execution identity in the notebook activity within the ADF pipeline.

please correct me if wrong
Saikumar Karri 0 Reputation points

2025-09-12T08:35:40.5366667+00:00

Hi Vrishabh,

Thanks for answering my question. And i have another question related to service principal and databricks,.

Is it possible to login to databricks workspace using service principal?

If yes, can you please let me know how can we do that ?

Thanks ,

Saikumar
Saikumar Karri 0 Reputation points

2025-09-12T10:44:35.3366667+00:00

Thanks for answering,

I mean can we open the Databricks UI workspace with service principal like we open with Microsoft entra ID ? And can you please help what all permissions required to get this ?
Anonymous

2025-09-12T11:41:59.8833333+00:00

Hi @Saikumar karri

Service principals in Azure Databricks are API-only identities. They are designed for automation and programmatic access, such as running jobs, accessing data, or managing resources via REST APIs, SDKs, or CLI. They cannot log into the Databricks UI like regular users authenticated via Microsoft Entra ID.

Kindly refer the below official document -

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals

Thanks,
Vrishabh
Saikumar Karri 0 Reputation points

2025-09-12T20:28:32.2633333+00:00

Hi @Vrishabhanath Patil

Thanks for answering, I still not able to find the service principal option in ADF while creating databricks linked service
Saikumar Karri 0 Reputation points

2025-09-13T16:47:47.77+00:00

Hi Vrishabh,

I have tried the provided steps to create access token but

I could not find a way to create a access token through service principal

Can you help me with detailed steps or documentation

Thanks

Saikumar
Anonymous

2025-09-14T13:57:40.1733333+00:00

Hi @Saikumar karri

Thank you for your patience and cooperation. I kindly request you to review the recommended official document provided below thoroughly.

https://learn.microsoft.com/en-us/azure/databricks/security/secrets/

After storing the secrets in Key Vault using the above document, please refer to the screenshot below to use those secrets when creating a linked service. Ensure you select the highlighted 'Authentication type' and provide the required Key Vault parameters.

Thanks,
Vrishabh
Anonymous

2025-09-15T11:50:50.5433333+00:00

Hi @Saikumar karri

Just checking in to confirm whether the steps provided above have resolved your query, or if you need any further assistance from our side.

Thanks,
Vrishabh
Saikumar Karri 0 Reputation points

2025-09-15T19:00:21.61+00:00

Hi @Vrishabhanath Patil ,

I logged in to Databricks using my email account, added the service principal, and assigned it cluster creation and workspace access permissions. After that, I generated a token, stored it in Key Vault, and used it as per the steps mentioned above. However, when I run the notebook, it still shows my user email instead of the service principal.

Thanks,

Saikumar
Anonymous

2025-09-18T10:59:28.29+00:00

Hi @Anonymous

Please confirm, if above given steps are helpful or do you need any further assistance.

Thanks,
Vrishabh

Answer 1

Saikumar Karri - Thanks for the question and using MS Q&A platform.

Here are the few important notes which needs to understand while setting up service principle with Azure Databricks:

Service Principals cannot log into the Databricks UI; they are meant for automation and API access only.
If you don’t see “Service Principal” as an option in ADF, use the Access Token method with a token generated via SP credentials.
Managed Identity is preferred for production due to its simplicity and security.

Questions on Databricks Service Principal Login and ADF Pipeline Notebook Execution Identity?

Azure Data Factory (ADF) does not currently support Service Principal (SP) authentication for Azure Databricks (ADB) linked services.

What does it means:
Service Principal cannot be used directly in ADF to authenticate with Databricks.

If you want to use a non-user identity, Managed Identity is the best option:
- Assign a System-Assigned or User-Assigned Managed Identity to ADF.
- Grant that identity appropriate roles in the Databricks workspace.
- Configure the ADF linked service to use Managed Identity.

Hope this helps. Let me know if you have any further questions or need additional assistance. Also, if these answers your query, do click the "Upvote" and click "Accept the answer" of which might be beneficial to other community members reading this thread.

𝘛𝘰 𝘴𝘵𝘢𝘺 𝘪𝘯𝘧𝘰𝘳𝘮𝘦𝘥 𝘢𝘣𝘰𝘶𝘵 𝘵𝘩𝘦 𝘭𝘢𝘵𝘦𝘴𝘵 𝘶𝘱𝘥𝘢𝘵𝘦𝘴 𝘢𝘯𝘥 𝘪𝘯𝘴𝘪𝘨𝘩𝘵𝘴 𝘰𝘯 𝘈𝘻𝘶𝘳𝘦 𝘋𝘢𝘵𝘢𝘣𝘳𝘪𝘤𝘬𝘴, 𝘥𝘢𝘵𝘢 𝘦𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨, 𝘢𝘯𝘥 Data & AI 𝘪𝘯𝘯𝘰𝘷𝘢𝘵𝘪𝘰𝘯𝘴, 𝘧𝘰𝘭𝘭𝘰𝘸 𝘮𝘦 𝘰𝘯 𝘓𝘪𝘯𝘬𝘦𝘥𝘐𝘯.

Answer 2

Hi @Saikumar karri

Thank you for reviewing the previously provided steps. Below are a few additional actions that may assist in addressing your concern -

Azure Databricks using a service principal for authentication, but the notebook execution still reflects your personal user identity. This typically happens when:

The token used for authentication is not being applied correctly. The Databricks workspace or notebook execution context is still tied to your user session. The service principal is not being used to initiate the actual API calls or cluster jobs.

Confirm the below steps -

Ensure the token you're using is generated for the service principal, not your user account. You can use OAuth 2.0 to authorize service principal access to Databricks resources via CLI or REST API.

Follow this guide: Authorize service principal access to Azure Databricks with OAuth

Avoid Manual UI Execution -

If you open and run the notebook manually in the Databricks UI, it will always show your email. To run as the service principal:

Use Databricks REST API or Databricks CLI with the service principal token.

Assign Proper Permissions Ensure the service principal has: Workspace access Cluster creation permissions Job execution permissions Reference: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/service-principals

Use Azure Data Factory (ADF) ADF does not explicitly list "Service Principal" as an authentication method. Instead: Use the Access Token option. Ensure the token is generated using service principal credentials.

When a notebook runs in Databricks, the identity it uses depends on who actually triggers the job—not who created the token. So, if you want the job to reflect the service principal's identity, it needs to be launched by the service principal itself using the API, CLI, or an automated tool—not manually through the user interface.

Thanks,
Vrishabh

Anonymous

2025-09-19T15:06:22.68+00:00

Hi @Saikumar karri

Just checking, if above given steps resolved your query or do you need any further assistance from our end.

Thanks,
Vrishabh