I want to install python package in databrikcs job clusters and how to include this utility is "ini" file

Rohit Boddu 466 Reputation points
2021-02-12T11:39:13.357+00:00

Hi Team,

How to install any python package in databricks jobs cluster ..

Requirement 1= and there is many 30 job clusters in my environment .. i dont want to install package individually in each job clusters is there any way to install package in all of job clusters in one go..

Requirement 2 = how to place this in "ini" file so whenever new cluster is created it automatically installs this utility ..

Thanks & Regards,
Rohit

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Saurabh Sharma 23,816 Reputation points Microsoft Employee
    2021-02-12T23:09:50.34+00:00

    @Rohit Boddu Thanks for using Microsoft Q&A !!

    1. If all of your clusters are in same data-bricks workspace then you can use global init script either using the databricks UI or using Rest API. Please refer to the documentation - Add a global init script using the UI and Global Init Scripts API for details.
      You can add Global Init Scripts from UI by going to your "Admin Console" and +Add to add your required scripts - (See screenshots below)

    67537-image.png
    67525-image.png

    Please note that Only admins can create global init scripts.

    1. In order to place "init" file for a specific cluster, you need to put init scripts in a DBFS directory accessible by your cluster and then update the path under Advanced Options of the cluster. Please find below gif for you reference. Please refer to the documentation.

    67563-cluster-init-script.gif

    ----------

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.