I want to install python package in databrikcs job clusters and how to include this utility is "ini" file

Rohit Boddu 461 Reputation points
2021-02-12T11:39:13.357+00:00

Hi Team,

How to install any python package in databricks jobs cluster ..

Requirement 1= and there is many 30 job clusters in my environment .. i dont want to install package individually in each job clusters is there any way to install package in all of job clusters in one go..

Requirement 2 = how to place this in "ini" file so whenever new cluster is created it automatically installs this utility ..

Thanks & Regards,
Rohit

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,711 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Saurabh Sharma 21,196 Reputation points Microsoft Employee
    2021-02-12T23:09:50.34+00:00

    @Rohit Boddu Thanks for using Microsoft Q&A !!

    1. If all of your clusters are in same data-bricks workspace then you can use global init script either using the databricks UI or using Rest API. Please refer to the documentation - Add a global init script using the UI and Global Init Scripts API for details.
      You can add Global Init Scripts from UI by going to your "Admin Console" and +Add to add your required scripts - (See screenshots below)

    67537-image.png
    67525-image.png

    Please note that Only admins can create global init scripts.

    1. In order to place "init" file for a specific cluster, you need to put init scripts in a DBFS directory accessible by your cluster and then update the path under Advanced Options of the cluster. Please find below gif for you reference. Please refer to the documentation.

    67563-cluster-init-script.gif

    ----------

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.