Share via

Workspace libraries (legacy)

  • Article

Important

This documentation has been retired and might not be updated. The products, services, or technologies mentioned in this content are no longer supported. See Cluster libraries.

Note

Workspace libraries have been deprecated and should not be used. However, storing libraries as workspace files is distinct from workspace libraries and is still fully supported. You can install libraries stored as workspace files directly to compute or job tasks.

Workspace libraries serve as a local repository from which you create cluster-installed libraries. A workspace library might be custom code created by your organization, or might be a particular version of an open-source library that your organization has standardized on.

You must install a workspace library on a cluster before it can be used in a notebook or job. See Install a workspace library onto a cluster.

Workspace libraries in the Shared folder are available to all users in a workspace, while workspace libraries in a user folder are available only to that user.

Important

Libraries can be installed from DBFS when using Databricks Runtime 14.3 LTS and below. However, any workspace user can modify library files stored in DBFS. To improve the security of libraries in a Azure Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default in Databricks Runtime 15.1 and above. See Storing libraries in DBFS root is deprecated and disabled by default.

Instead, Databricks recommends uploading all libraries, including Python libraries, JAR files, and Spark connectors, to workspace files or Unity Catalog volumes, or using library package repositories. If your workload does not support these patterns, you can also use libraries stored in cloud object storage.

How do workspace libraries differ from libraries stored with workspace files?

The term “workspace libraries” refers to any library registered to a workspace. Workspace libraries appear alongside notebooks and workspace files and follow ACL rules for their containing directory.

The term “workspace files” refers to any arbitrary file stored in the workspace. You can store Python .whl packages as workspace files and then register them as workspace libraries. See What are workspace files?.

You can upload libraries as workspace files. See Workspace files basic usage.

While you can upload many file types to workspace files, only Python .whl files can be installed using workspace files. See Cluster-scoped libraries.

In Databricks Runtime 13.3 LTS and above, you can use %pip to install Python wheel files stored as workspace files. See Install a package stored as a workspace file with %pip.

Create a workspace library

Warning

This functionality is no longer available. See Creation and installation of workspace libraries is no longer available.

  1. Right-click the workspace folder where you want to store the library.
  2. Select Create > Library. The Create Library dialog appears.
  3. Select the Library Source and follow the appropriate procedure:

Upload a Jar, Python egg, or Python wheel file

Note

Installing Python egg files is deprecated and will be removed in a future Databricks Runtime release.

Important

Libraries can be installed from DBFS when using Databricks Runtime 14.3 LTS and below. However, any workspace user can modify library files stored in DBFS. To improve the security of libraries in a Azure Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default in Databricks Runtime 15.1 and above. See Storing libraries in DBFS root is deprecated and disabled by default.

Instead, Databricks recommends uploading all libraries, including Python libraries, JAR files, and Spark connectors, to workspace files or Unity Catalog volumes, or using library package repositories. If your workload does not support these patterns, you can also use libraries stored in cloud object storage.

  1. In the Library Source button list, select Upload.
  2. Select Jar, Python Egg, or Python Whl.
  3. Optionally enter a library name.
  4. Drag your Jar, Egg, or Whl to the drop box or click the drop box and navigate to a file. The file is uploaded to dbfs:/FileStore/jars.
  5. Click Create. The library status screen displays.

Reference an uploaded JAR, Python egg, or Python wheel file

You can create a new workspace library by referencing JAR, Python egg, or Python wheel files stored in the DBFS root, on object storage, or with workspace files. Following this workflow registers a pointer to a library without uploading or moving files.

With Databricks Runtime 13.2 and above, you can also reference a volume in Unity Catalog. This capability is in Public Preview.

For shared access mode, you must add JARs to the allowlist. See Allowlist libraries and init scripts on shared compute.

  1. Select File path/ADLS in the Library Source button list.
  2. Select Jar, Python Egg, or Python Whl.
  3. Optionally enter a library name.
  4. Specify the path to the library, such as the following examples:
    • abfss://container-name@storage-account-name.dfs.core.windows.net/path/to/library.whl
    • /Workspace/path/to/library.whl
    • dbfs:/path/to/library.whl
    • /Volumes/<catalog>/<schema>/<volume>/<path_to_library_file>
  5. Click Create. The library status screen displays.

For more on working with libraries in object storage, see Install libraries from object storage.

For more on workspace files, see Work with files on Azure Databricks.

Install a workspace library onto a cluster

Note

Azure Databricks processes all workspace libraries in the order that they were installed on the cluster. You might need to pay attention to the order of installation on the cluster if there are dependencies between libraries.

Important

Libraries can be installed from DBFS when using Databricks Runtime 14.3 LTS and below. However, any workspace user can modify library files stored in DBFS. To improve the security of libraries in a Azure Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default in Databricks Runtime 15.1 and above. See Storing libraries in DBFS root is deprecated and disabled by default.

Instead, Databricks recommends uploading all libraries, including Python libraries, JAR files, and Spark connectors, to workspace files or Unity Catalog volumes, or using library package repositories. If your workload does not support these patterns, you can also use libraries stored in cloud object storage.

In single user access mode, the identity of the assigned principal (a user or service principal) is used. In shared access mode, libraries use the identity of the user who installed the library. No-isolation shared access mode does not support volumes, but uses the same identity assignment as shared access mode.

To see all options for installing libraries on clusters, see Cluster libraries.

To install a library that already exists in the workspace, you can start from the cluster UI or the library UI:

Cluster

  1. In the sidebar, click compute icon Compute.
  2. Click a cluster name.
  3. Click the Libraries tab.
  4. Click Install New.
  5. In the Library Source button list, select Workspace Library.
  6. Select a workspace library.
  7. Click Install.

Library

  1. Go to the folder containing the library.
  2. Click the library name.
  3. Select the checkbox next to the cluster that you want to install the library on and click Install. Only running clusters appear in the list.

View workspace library details

Note

Workspace folders provide convenience for discovering workspace libraries and managing ACLs.

  1. Go to the workspace folder containing the library.
  2. Click the library name.

The library details page shows the running clusters and the install status of the library. If the library is installed, the page contains a link to the package host. If the library was uploaded, the page displays a link to the uploaded package file.

Move a workspace library

Note

Workspace folders provide convenience for discovering workspace libraries and managing ACLs. Moving a workspace library does not move files, but can modify which users have access to the workspace library.

  1. Go to the workspace folder containing the library.
  2. Right-click the library name and select Move. A folder browser displays.
  3. Click the destination folder.
  4. Click Move.

Delete a workspace library

Important

Before deleting a workspace library, you should uninstall it from all clusters.

To delete a workspace library:

  1. Move the library to the Trash folder.
  2. Either permanently delete the library in the Trash folder or empty the Trash folder.