Configure default package repositories

Important

Configuring default Python package repositories is generally available for serverless notebooks and jobs. For Lakeflow pipelines and for classic compute created via the UI and API, it is in Public Preview. For Databricks Apps, it is in Beta. Workspace admins can control access to these features from the Previews page by enabling the following previews:

  • Default Python package repositories in Spark Declarative Pipelines (Public Preview)
  • Default Python package repositories in clusters created via API (Public Preview)
  • Default Python package repositories in clusters created via UI (Public Preview)
  • Enable Workspace Repo Configuration for Databricks Apps (Beta)

Workspace admins can configure private or authenticated package repositories within workspaces as the default package repositories for notebooks, jobs, Lakeflow pipelines, and Databricks Apps. Python repositories apply to all of these workloads, but npm registries apply to Databricks Apps only.

If a workspace is configured with a default package repository, users in the workspace can install packages from internal repositories without explicitly defining index-url or extra-index-url values for Python, or a registry for npm. If these values are specified in code, in a notebook, or in an app's app.yaml file, they take precedence over the workspace defaults. See Use workspace-level default package repositories for how precedence works for Databricks Apps.

This configuration leverages Databricks secrets to securely store and manage repository URLs and credentials. Workspace admins can configure the setup using the workspace admin settings page or using a predefined secret scope and the Databricks CLI secrets commands or the REST API.

Set up default dependencies for a workspace

Workspace admins can add or remove the default package repositories using the workspace admin settings page.

  1. As a workspace administrator, log in to the Databricks workspace.
  2. Click your username in the top bar of the Databricks workspace and select Settings.
  3. Click on the Compute tab.
  4. Next to Default Package Repositories, click Manage.
  5. (Optional) Add or remove a Python index URL, extra index URLs, a custom SSL certificate, or an npm registry configuration.
  6. Click Save to save the changes.

Note

Modifications or deletions to secrets are applied differently depending on the workload type. For serverless notebooks and jobs, changes are applied after reattaching the compute resource to the notebook or rerunning the job. For classic notebooks and jobs, changes are applied after the compute resource is restarted. For Lakeflow pipelines, changes are applied to new pipeline runs. For Databricks Apps, changes are applied the next time the app is built and deployed.

Set up using the secrets CLI or REST API

To configure default package repositories using the CLI or REST API, create a predefined secret scope and configure access permissions, then add the package repository secrets.

Predefined secret scope name

Workspace administrators can set default pip index URLs, extra index URLs, an SSL certificate, and an npm configuration, along with authentication tokens and secrets, in a designated secret scope under predefined keys:

  • Secret scope name: databricks-package-management
  • Secret key for index-url: pip-index-url
  • Secret key for extra-index-urls: pip-extra-index-urls
  • Secret key for SSL certification content: pip-cert
  • Secret key for npm configuration: npm-rc (Databricks Apps only)

Create the secret scope

A secret scope can be created using the Databricks CLI secrets commands or the REST API. After creating the secret scope, configure access control lists to grant all workspace users read access. This ensures that the repository remains secure and cannot be altered by individual users. The secret scope must use the predefined secret scope name databricks-package-management.

databricks secrets create-scope databricks-package-management
databricks secrets put-acl databricks-package-management admins MANAGE
databricks secrets put-acl databricks-package-management users READ

Add package repository secrets

Add package repository details using the predefined secret key names. Python repositories apply to notebooks, jobs, Lakeflow pipelines, and Databricks Apps, but npm registries apply to Databricks Apps only.

Python

Add the Python package repository details using the predefined secret key names, with all three fields being optional.

# Add index URL.
databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-index-url", "string_value":"<index-url-value>"}'

# Add extra index URLs. If you have multiple extra index URLs, separate them using white space.
databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-extra-index-urls", "string_value":"<extra-index-url-1 extra-index-url-2>"}'

# Add cert content. If you want to pip configure a custom SSL certificate, put the cert file content here.
databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-cert", "string_value":"<cert-content>"}'

Note

If your repository requires authentication, use one of the following formats for the index URL:

  • Username and password: https://<username>:<password>@<index_url>
  • Token: https://<token>@<index_url>

Node.js

For Databricks Apps, you can also configure a default npm registry so that apps install Node.js packages, including private packages, from the workspace-configured registry. Store the full .npmrc file content, including the registry URL and any authentication token, under the npm-rc secret key:

databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "npm-rc", "string_value":"registry=https://npm.example.com/\n//npm.example.com/:_authToken=<token>\n"}'

When an app in the workspace is built, Azure Databricks writes this content to the app's .npmrc file. For how precedence works and how to override it for a specific app, see Node.js packages.

Modify or delete package repository secrets

To modify package repository secrets, use the put-secret command. To delete package repository secrets, use delete-secret as shown below:

# delete secret
databricks secrets delete-secret databricks-package-management pip-index-url

databricks secrets delete-secret databricks-package-management pip-extra-index-urls

databricks secrets delete-secret databricks-package-management pip-cert

# delete the npm configuration secret (Databricks Apps only)
databricks secrets delete-secret databricks-package-management npm-rc

# delete scope
databricks secrets delete-scope databricks-package-management
  • Last updated on