To make third-party or custom code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.
Azure Databricks includes many common libraries in Databricks Runtime. To see which libraries are included in Databricks Runtime, look at the System Environment subsection of the Databricks Runtime release notes for your Databricks Runtime version.
Azure Databricks does not invoke Python
atexit functions when your notebook or job completes processing. If you use a Python library that registers
atexit handlers, you must ensure your code calls required functions before exiting.
Installing Python eggs is deprecated and will be removed in a future Databricks Runtime release. Use Python wheels or install packages from PyPI instead.
Microsoft Support helps isolate and resolve issues related to libraries installed and maintained by Azure Databricks. For third-party components, including libraries, Microsoft provides commercially reasonable support to help you further troubleshoot issues. Microsoft Support assists on a best-effort basis and might be able to resolve the issue. For open source connectors and projects hosted on Github, we recommend that you file issues on Github and follow up on them. Development efforts such as shading jars or building Python libraries are not supported through the standard support case submission process: they require a consulting engagement for faster resolution. Support might ask you to engage other channels for open-source technologies where you can find deep expertise for that technology. There are several community sites; two examples are the Microsoft Q&A page for Azure Databricks and Stack Overflow.
You can install libraries in three modes: workspace, cluster-installed, and notebook-scoped.
- Workspace libraries serve as a local repository from which you create cluster-installed libraries. A workspace library might be custom code created by your organization, or might be a particular version of an open-source library that your organization has standardized on.
- Cluster libraries can be used by all notebooks running on a cluster. You can install a cluster library directly from a public repository such as PyPI or Maven, or create one from a previously installed workspace library.
- Notebook-scoped libraries, available for Python and R, allow you to install libraries and create an environment scoped to a notebook session. These libraries do not affect other notebooks running on the same cluster. Notebook-scoped libraries do not persist and must be re-installed for each session. Use notebook-scoped libraries when you need a custom environment for a specific notebook.
This section covers:
The following table provides an overview of options you can use to install Python libraries in Azure Databricks.
- Custom containers that use a conda-based environment are not compatible with notebook-scoped libraries in Databricks Runtime 9.0 and above and with cluster libraries in Databricks Runtime 10.1 and above. Instead, Azure Databricks recommends installing libraries directly in the image or using init scripts. To continue using cluster libraries in those scenarios, you can set the Spark configuration
false. Support for the Spark configuration will be removed on or after December 31, 2021.
- Notebook-scoped libraries using magic commands are enabled by default. See Requirements for details.
- Notebook-scoped libraries with the library utility are deprecated.
|Python package source||Notebook-scoped libraries with %pip||Notebook-scoped libraries with the library utility (deprecated)||Cluster libraries||Job libraries with Jobs API|
||Select PyPI as the source.||Add a new
|Private PyPI mirror, such as Nexus or Artifactory||Use
||Not supported.||Not supported.|
|VCS, such as GitHub, with raw source||Use
||Not supported.||Select PyPI as the source and specify the repository URL as the package name.||Add a new
|Private VCS with raw source||Use
||Not supported.||Not supported.||Not supported.|
||Select DBFS as the source.||Add a new