Python packages and storage location

Question

All,

I downloaded whl file for folium and uploaded into the workspace. I also imported into the workspace whoosh directly from pypi. I created a cluster and I loaded great_expectations module into the cluster (not the workspace).

I can see that the folium whl file is stored in dbfs "dbfs:/FileStore/jars/15c44e71_0fee_464b_8fa1_5c81dc48b234/folium-0.12.1.post1-py2.py3-none-any.whl". But I cannot see where whoosh is stored though I loaded it into the workspace and I also cannot see where great_expectations library is stored which I uploaded it into the cluster.

Answer 1

Hello @grajee ,

Thanks for the question and using MS Q&A platform.

Unfortunately, user doesn't have access to the libraries installed directly from a public repository such as PyPI or Maven.

Note: Cluster libraries can be used by all notebooks running on a cluster. You can install a cluster library directly from a public repository such as PyPI or Maven, or create one from a previously installed workspace library.

There are two primary ways to install a library on a cluster:

• Install a workspace library that has been already been uploaded to the workspace.
• Install a library for use with a specific cluster only.

For more details, refer to Azure Databricks - Libraries.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators