How to create a global init script to install an .whl ​​library on all clusters?

Question

Hello,

I created a globat init script to install the package on all clusters at startup. The script runs successfully and I can import the functions from the package, but the library installed doesn't appear in the Libraries tab of the cluster. Am I doing something wrong?

• Script

• Runtime

• Logs

• Libraries

No library is displayed. This is a problem because those who use the cluster will not know that the library has been installed.

References:

• https://learn.microsoft.com/en-us/azure/databricks/clusters/init-scripts#global-init-scripts
• https://github.com/MicrosoftDocs/azure-docs/issues/76752

Best regards,
Cristina

Answer 1

Hello @Cristina Santana Souza ,

Thanks for the question and using MS Q&A platform.

Unfortunately, libraries installed using Global/Init scripts does not show in libraries section.

Note: Even, if you install libraries using Notebooks even those will not be shown in the libraries section in the cluster.

Example: I had installed a library called openpyxl via Global/Init/Notebook does not show in libraries section in the cluster.

Global Init Script:

Event Log confirming the INIT_SCRIPTS_FINISHED:

Cluster shows: No library is displayed.

Successfully able to use the installed library named openpyxl in the notebooks:

How to verify whether package is installed or not?

You can use the below command to get libraries that are installed:

import pkg_resources  
for d in pkg_resources.working_set:  
     print(d)  

First lets verify the installed version of openpyxl on the cluster.

import pkg_resources  
pkg_resources.get_distribution('openpyxl').version  

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.