Run a job with a Microsoft Entra ID service principal

Lakeflow Jobs provide a non-interactive way to run applications in an Azure Databricks cluster, for example, an ETL job or data analysis task that runs on a scheduled basis. Typically, these jobs run as the user who created them, but there are some limitations:

• Creating and running jobs is dependent on the user having appropriate permissions.
• The user might be removed from the Azure Databricks workspace, causing the job to fail.

To address these limitations, configure your job to Run as a Microsoft Entra ID service principal — a service identity associated with an application rather than a specific user. The job runs with the service principal's identity and permissions, regardless of who triggers it. See Configure the Run as user for job runs for more on the Run as setting.

For example, when service principals control access to data stored in an Azure Data Lake Storage account, configure a job to Run as a service principal with access to that storage account so it can read and write the data without depending on any individual user's permissions.

This tutorial walks you through how to configure a job to Run as a Microsoft Entra ID service principal:

1. Create a service principal in Microsoft Entra ID.
2. Add the service principal to the Azure Databricks workspace.
3. Grant yourself the Service Principal User role on the service principal.
4. (Optional) Configure Azure Data Lake Storage access for the service principal using a secret scope.
5. Create the job and configure it to Run as the service principal.
6. Test the job.

If you don't have an Azure subscription, create a free account before you begin.

Note

You cannot use a cluster with credential passthrough enabled to run a job as a service principal. If your job requires a service principal to access Azure storage, see Connect to Azure Data Lake Storage or Blob Storage using Azure credentials.

Requirements

You'll need the following for this tutorial:

• A user account with the permissions required to register an application in your Microsoft Entra ID tenant.
• Workspace admin or account admin privileges in the Azure Databricks workspace where you'll run jobs (required to add a service principal to the workspace).
• The Service Principal Manager role on the service principal. The creator of a service principal automatically has this role. See Roles for managing service principals.

Step 1: Create a service principal in Microsoft Entra ID

A service principal is the identity of a Microsoft Entra ID application. To create the service principal that runs jobs:

1. In the Azure portal, select Microsoft Entra ID > App Registrations > New Registration. Enter a name for the application and click Register.
2. Go to Overview and note the Application (client) ID and Directory (tenant) ID.

If your job needs to authenticate to other Azure resources (for example, to read from Azure Data Lake Storage) using the service principal's credentials, also generate a client secret:

1. Go to Certificates and secrets, click New client secret, and generate a new client secret. Copy and save the secret in a secure place.

Step 2: Add the service principal to the Azure Databricks workspace

You can add the service principal to your workspace using the workspace administrator settings page or the account console. The following steps use the workspace administrator settings path:

1. As a workspace administrator, log in to the Azure Databricks workspace.
2. Click your username in the top bar and select Settings.
3. Click the Identity and access tab.
4. Next to Service principals, click Manage.
5. Click Add service principal.
6. Click Add new.
7. For Management, select Microsoft Entra ID managed, and paste the Application (client) ID from Step 1.
8. Enter a name for the service principal (for example, test-sp).
9. Click Add.

For other ways to add a service principal — including the account console flow and the automatic identity management option — see Add service principals to your account.

After you add the service principal to the workspace, give it the entitlements it needs to run jobs (for example, Allow unrestricted cluster creation or Allow cluster creation) on the same Service principals page.

Step 3: Grant yourself the Service Principal User role on the service principal

To configure a job to Run as the service principal, you must have the Service Principal User role on it. The creator of a service principal automatically gets the Service Principal Manager role, but the Service Principal User role does not transfer automatically — you must grant it explicitly.

1. In the Azure Databricks workspace, click your username in the top bar and select Settings.
2. Click the Identity and access tab.
3. Next to Service principals, click Manage.
4. Click the service principal you added in Step 2.
5. Click the Permissions tab.
6. Click Grant access.
7. Search for and select your user account.
8. Select Service principal: User.
9. Click Save.

For more information on service principal roles, see Roles for managing service principals.

Step 4 (optional): Configure Azure Data Lake Storage access for the service principal

If your job needs to read from or write to Azure Data Lake Storage using the service principal's credentials, configure a secret scope to securely hold the service principal's client secret, and add Spark configuration to the job cluster so it can authenticate to storage.

If your job doesn't need Azure Data Lake Storage access via the service principal, skip to Step 5.

Create an Azure Key Vault-backed secret scope in Azure Databricks

Manage secret scopes provide secure storage and management of secrets. You can store the service principal's client secret in either a Azure Databricks-backed scope or an Azure Key Vault-backed scope. These instructions describe the Azure Key Vault-backed option:

1. Create an Azure Key Vault instance in the Azure portal:

   1. In the Azure portal, select Key Vaults > + Add and give the Key Vault a name.

   2. Click Review + create.

   3. After validation completes, click Create.

   4. Open the new Key Vault and go to the Properties page.

   5. Copy and save the Vault URI and Resource ID.

      

2. Create the Azure Databricks secret scope backed by the Azure Key Vault instance:

   1. Go to the Azure Databricks Create Secret Scope page at https://<per-workspace-url>/#secrets/createScope. Replace <per-workspace-url> with the unique per-workspace URL for your Azure Databricks workspace.

   2. Enter a Scope Name.

   3. Enter the Vault URI and Resource ID values for the Azure Key Vault you created.

   4. Click Create.

      

Save the client secret in Azure Key Vault

1. In the Azure portal, go to the Key vaults service.

2. Select the Key Vault you created.

3. Under Settings > Secrets, click Generate/Import.

4. Select the Manual upload option and enter the client secret in the Value field.

   

5. Click Create.

Grant the service principal read access to the secret scope

Open a terminal and use the Databricks CLI to run the following command:

databricks secrets put-acl <scope-name> <application-id> READ

• Replace <scope-name> with the name of the Azure Databricks secret scope that contains the client secret.
• Replace <application-id> with the Application (client) ID for the Microsoft Entra ID application registration.

Step 5: Create the job

You're now ready to create a job. You'll use a notebook created in the Azure Databricks UI.

1. Go to your Azure Databricks landing page and select New > Notebook. Give your notebook a name and select SQL as the default language.
2. Enter SELECT 1 in the first cell of the notebook. This is a simple command that just displays 1 if it succeeds. If you have granted your service principal access to particular files or paths in Azure Data Lake Storage, you can read from those paths instead.
3. In the workspace sidebar, click Jobs and Pipelines.
4. Click Create, then Job.
5. Click the Notebook tile to configure the first task. If the Notebook tile is not available, click Add another task type and search for Notebook.
6. Enter a Task name.
7. Select the notebook you just created for the Path field.

If you completed the optional Step 4 and need cluster configuration for Azure Data Lake Storage OAuth access, click Edit next to the cluster, click Advanced Options, and on the Spark tab enter the following Spark Config:

fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}
fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token

• Replace <storage-account> with the name of the storage account containing your data.
• Replace <secret-scope-name> with the name of the Azure Databricks secret scope that contains the client secret.
• Replace <application-id> with the Application (client) ID for the Microsoft Entra ID application registration.
• Replace <secret-name> with the name associated with the client secret value in the secret scope.
• Replace <directory-id> with the Directory (tenant) ID for the Microsoft Entra ID application registration.

Click Create task.

Step 6: Configure the job to Run as the service principal

A job's Run as setting controls the identity that the job runs as. By default, this is the job's creator. To configure the job to Run as the service principal:

1. In the workspace sidebar, click Jobs and Pipelines, and click the job you created.
2. In the Job details side pane, click the pencil icon next to the Run as field.
3. Search for and select the service principal you added in Step 2.
4. Click Save.

For more information on the Run as setting and how it interacts with Unity Catalog permissions and SQL task identities, see Manage identities, permissions, and privileges for Lakeflow Jobs.

Step 7: Test the job

You run jobs configured with Run as the same way you run jobs as a user, either through the UI, API, or CLI. To test the job using the Azure Databricks UI:

1. Go to Jobs & Pipelines in the Azure Databricks UI and select the job.
2. Click Run now.

You see a status of Succeeded for the job if everything runs correctly. The job runs with the service principal's identity, regardless of who clicks Run now. Select the job to verify the output:

Learn more

• Lakeflow Jobs
• Manage identities, permissions, and privileges for Lakeflow Jobs
• Service principals
• Roles for managing service principals