Different Version in cluster and workspace

Question

Different Version in cluster and workspace

Krishnamoorthy, Rajamannar 61

I have installed NumPy version 1.20.0 in my cluster by when I check the same in the cluster it's a different version 1.19.5. I have attached both the cluster and workspace screenshots. Also installing the packages in the cluster libraries takes more than 5 minutes but the scheduled job starts before the packages are installed, is there any way to speed up the package installation process or start the job after pkg installation.

Cluster Version

Workspace Version

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-10-29T10:01:47.923+00:00
Hello @Krishnamoorthy, Rajamannar ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

Accepted answer

1 additional answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-10-29T10:01:47.923+00:00

Hello @Krishnamoorthy, Rajamannar ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

Hello @Krishnamoorthy, Rajamannar ,

Welcome to the Microsoft Q&A platform.

Make sure to restart the cluster to reflect the new packages that you have installed.

As per the repro, I had used numpy==1.20.0 package installation. By default it comes with 1.19.2 version and after installing 1.20.0 version you need to restart the cluster to reflect the newer package versions.

Regarding the packages in the cluster libraries takes more than 5 minutes this is an excepted behaviour based on the package which you are trying to install.

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Krishnamoorthy, Rajamannar 61 Reputation points

2021-10-29T12:43:50.743+00:00

After installing I did restart my cluster but still, the NumPy remained unchanged
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-11-03T10:18:03.727+00:00

Hello @Krishnamoorthy, Rajamannar ,

Could you please help with the Azure Databricks Runtime which are using?
Krishnamoorthy, Rajamannar 61 Reputation points

2021-11-08T05:45:56.753+00:00

Hello @PRADEEPCHEEKATLA
I'm using 9.0 ML runtime
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-11-08T09:54:21.66+00:00

Hello @Krishnamoorthy, Rajamannar ,

I had tested on Azure Databricks Runtime: 9.0 ML and works as excepted.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-11-09T12:23:15.103+00:00

Hello @Krishnamoorthy, Rajamannar ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

Answer 2

Shalvin 161

Hello @Krishnamoorthy, Rajamannar ,

I also have seen similar observation and the solution was to install packages when I wanted. Below steps has worked for me always.

Create a requirements.txt file with the package versions you want
Upload the file to DBFS (Eg: /FileStore/requirements.txt)
Use one of the below options to load the requirements

i. Create a cell as the first code cell in your job notebook and run the command %pip install -r /dbfs/FileStore/requirements.txt (I use this approach)
ii. Use a cluster init script to run the command pip install -r /dbfs/FileStore/requirements.txt

Thanks,
Shalvin

Krishnamoorthy, Rajamannar 61 Reputation points

2021-10-31T02:06:17.817+00:00

Hello @Shalvin ,

Thanks for answering.
In the first approach installed packages will be on notebook scope right but I want the packages to be on cluster level.

Can you provide any sample python script on cluster init script? I went through your referred link but still come up with a solution.
Also, Say if I use the init scripts will the packages be installed first and then the cluster is started or its the other way?

Thanks,
Rajamannar
Shalvin 161 Reputation points

2021-10-31T15:58:35.027+00:00
Hello @Krishnamoorthy, Rajamannar ,

There is nothing like 'Notebook scope' as far as understand. The package version you install will be there until you install via another pip install or do a cluster restart. Cluster will be Linux and hence a bash script will work. Please use the following in the file /dbfs/FileStore/init.sh and that should work.

pip install -r ./requirements.txt

Though unrelated, the recommended approach for running job is using an on demand job cluster which will be cost effective and at the same time isolate. Only draw back is, it will take some 3-5 minutes for the job to actually start as it needs to build the cluster first. If the job is a longer one, you could try that as well.

Thanks,
Shalvin

Share via

Different Version in cluster and workspace

1 additional answer

Your answer