Workspace libraries

  • Article

Workspace libraries serve as a local repository from which you create cluster-installed libraries. A workspace library might be custom code created by your organization, or might be a particular version of an open-source library that your organization has standardized on.

You must install a workspace library on a cluster before it can be used in a notebook or job. See Install a workspace library onto a cluster.

Workspace libraries in the Shared folder are available to all users in a workspace, while workspace libraries in a user folder are available only to that user.

Important

Libraries uploaded using the library UI are stored in the DBFS root. All workspace users have the ability to modify data and files stored in the DBFS root. You can avoid this by uploading libraries to workspace files or Unity Catalog volumes, using libraries in cloud object storage or using library package repositories.

How do workspace libraries differ from libraries stored with workspace files?

The term “workspace libraries” refers to any library registered to a workspace. Workspace libraries appear alongside notebooks and workspace files and follow ACL rules for their containing directory.

The term “workspace files” refers to any arbitrary file stored in the workspace. You can store Python .whl packages as workspace files and then register them as workspace libraries. See What are workspace files?.

You can upload libraries as workspace files. See Workspace files basic usage.

While you can upload many file types to workspace files, only Python .whl files can be installed using workspace files. See Compute compatibility with libraries and init scripts.

In Databricks Runtime 13.3 LTS and above, you can use %pip to install Python wheels stored as workspace files. See Install a package stored as a workspace file with %pip.

Create a workspace library

  1. Right-click the workspace folder where you want to store the library.
  2. Select Create > Library. The Create Library dialog appears.
  3. Select the Library Source and follow the appropriate procedure:

Upload a Jar, Python egg, or Python wheel

Note

Installing Python eggs is deprecated and will be removed in a future Databricks Runtime release.

Important

Libraries uploaded using the library UI are stored in the DBFS root. All workspace users have the ability to modify data and files stored in the DBFS root. You can avoid this by uploading libraries to workspace files or Unity Catalog volumes, using libraries in cloud object storage or using library package repositories.

  1. In the Library Source button list, select Upload.
  2. Select Jar, Python Egg, or Python Whl.
  3. Optionally enter a library name.
  4. Drag your Jar, Egg, or Whl to the drop box or click the drop box and navigate to a file. The file is uploaded to dbfs:/FileStore/jars.
  5. Click Create. The library status screen displays.

Reference an uploaded JAR, Python egg, or Python wheel

You can create a new workspace library by referencing JAR, Python egg, or Python wheel files stored in the DBFS root, on object storage, or with workspace files. Following this workflow registers a pointer to a library without uploading or moving files.

With Databricks Runtime 13.2 and above, you can also reference a volume in Unity Catalog. This capability is in Public Preview.

For shared access mode, you must add JARs to the allowlist. See Allowlist libraries and init scripts on shared compute.

  1. Select File path/ADLS in the Library Source button list.
  2. Select Jar, Python Egg, or Python Whl.
  3. Optionally enter a library name.
  4. Specify the path to the library, such as the following examples:
    • abfss://container-name@storage-account-name.dfs.core.windows.net/path/to/library.whl
    • /Workspace/path/to/library.whl
    • dbfs:/path/to/library.whl
    • /Volumes/<catalog>/<schema>/<volume>/<path_to_library_file>
  5. Click Create. The library status screen displays.

For more on working with libraries in object storage, see Install libraries from object storage.

For more on workspace files, see How to work with files on Azure Databricks.

Install a workspace library onto a cluster

Note

Azure Databricks processes all workspace libraries in the order that they were installed on the cluster. You might need to pay attention to the order of installation on the cluster if there are dependencies between libraries.

Important

Libraries uploaded using the library UI are stored in the DBFS root. All workspace users have the ability to modify data and files stored in the DBFS root. You can avoid this by uploading libraries to workspace files or Unity Catalog volumes, using libraries in cloud object storage or using library package repositories.

In assigned access mode, the identity of the assigned principal (a user or service principal) is used.

In shared access mode, libraries use the identity of the user who installed the library.

Note

No-isolation shared access mode does not support volumes, but uses the same identity assignment as shared access mode.

To see all options for installing libaries on clusters, see Cluster libraries.

To install a library that already exists in the workspace, you can start from the cluster UI or the library UI:

Cluster

  1. In the sidebar, click compute icon Compute.
  2. Click a cluster name.
  3. Click the Libraries tab.
  4. Click Install New.
  5. In the Library Source button list, select Workspace Library.
  6. Select a workspace library.
  7. Click Install.

Library

  1. Go to the folder containing the library.
  2. Click the library name.
  3. Select the checkbox next to the cluster that you want to install the library on and click Install. Only running clusters appear in the list.

View workspace library details

Note

Workspace folders provide convenience for discovering workspace libraries and managing ACLs.

  1. Go to the workspace folder containing the library.
  2. Click the library name.

The library details page shows the running clusters and the install status of the library. If the library is installed, the page contains a link to the package host. If the library was uploaded, the page displays a link to the uploaded package file.

Move a workspace library

Note

Workspace folders provide convenience for discovering workspace libraries and managing ACLs. Moving a workspace library does not move files, but can modify which users have access to the workspace library.

  1. Go to the workspace folder containing the library.
  2. Right-click the library name and select Move. A folder browser displays.
  3. Click the destination folder.
  4. Click Move.

Delete a workspace library

Important

Before deleting a workspace library, you should uninstall it from all clusters.

To delete a workspace library:

  1. Move the library to the Trash folder.
  2. Either permanently delete the library in the Trash folder or empty the Trash folder.