Manage identities, permissions, and privileges for pipelines

Identities, permissions, and privileges control who can run, manage, and query pipelines and the data they produce.

Databricks recommends using Unity Catalog for all new pipelines. By default, materialized views and streaming tables created by pipelines configured with Unity Catalog can only be queried by the pipeline owner. See Use Unity Catalog with pipelines.

If your pipelines publish datasets to legacy Hive metastore, see Use Lakeflow Spark Declarative Pipelines with legacy Hive metastore.

For general best practices on identity configurations, see Identity best practices.

What identity is used for pipeline updates?

Pipelines process updates using the identity of the pipeline owner. Assign a new pipeline owner to change the identity used to run the pipeline. See Change the pipeline owner.

Databricks recommends setting a service principal as the pipeline owner. See Service principals.

Who can run a pipeline update?

Pipeline updates can be run by any user or service principal with CAN RUN, CAN MANAGE, or IS OWNER permissions.

Who can view a pipeline and its output?

To open a pipeline and view its details, a user needs at least the CAN VIEW permission on the pipeline. For the full list of pipeline permission levels and the abilities each one grants, see Lakeflow Spark Declarative Pipelines ACLs.

To view the pipeline that backs a streaming table or materialized view, a non-admin user also needs the REFRESH privilege on that streaming table or materialized view, in addition to their permissions on the pipeline. Without the REFRESH privilege, the pipeline URL shows Pipeline not available.

Configure pipeline permissions

You must have the CAN MANAGE or IS OWNER permission on the pipeline to manage permissions. Pipelines use access control lists (ACLs) to control permissions. For a complete list of permissions and their abilities, see Lakeflow Spark Declarative Pipelines ACLs.

1. In the sidebar, click Jobs & Pipelines.
2. Select the Name of a pipeline.
3. Click Share. The Permissions Settings dialog appears.
4. Click Select User, Group or Service Principal… and select a user, group, or service principal.
5. Select a permission from the permission drop-down menu.
6. Click Add.
7. Click Save.

Change the pipeline owner

The pipeline owner is the identity that pipeline updates run as. Changing the owner changes the identity used for future updates.

To change a pipeline's owner, you must be both a metastore admin and a workspace admin. Change the owner using either the UI or the REST API.

Use the UI

1. In the sidebar, click Jobs & Pipelines.
2. Select the Name of the pipeline.
3. Click Share. The Permissions Settings dialog appears.
4. Clear the current owner, then select the new owner. The owner can be a user or a service principal. Databricks recommends a service principal. See Service principals.
5. Click Save.

Use the REST API

If the owner control is unavailable in the UI, such as for some internally managed pipelines, change the owner with the Set pipeline permissions REST API operation. Specify the new owner's user_name (or service_principal_name for a service principal) with the IS_OWNER permission level:

{
  "access_control_list": [
    {
      "user_name": "new.owner@example.com",
      "permission_level": "IS_OWNER"
    }
  ]
}

If no user is both a metastore admin and a workspace admin

If no one in your organization is both a metastore admin and a workspace admin, contact your Databricks representative to change the pipeline owner.

Allow non-admin users to view the driver logs from a Unity Catalog-enabled pipeline

By default, only the pipeline owner and workspace admins can view the driver logs from the cluster that runs a Unity Catalog-enabled pipeline. You can enable access to the driver logs for any user with CAN MANAGE, CAN VIEW, or CAN RUN permissions by adding the following Spark configuration parameter to the configuration object in the pipeline settings:

{
  "configuration": {
    "spark.databricks.acl.needAdminPermissionToViewLogs": "false"
  }
}