init script capture the Libraries

Question

init script capture the Libraries

Sharukh Kundagol 145

I Have requirment to capture all the py libraries in a files as part of init script. i have below command in init script.

pip freeze > /dbfs/databricks/scripts/library_versions.txt

but getting error : /dbfs/databricks/scripts/new_libraries-init.sh: 8: cannot create /dbfs/databricks/scripts/library_versions.txt : Invalid argument This error will not come when i use dbutils.fs.put to create the sh file, but when i deploy the script part of devops or copy the sh file from adls to dbfs this error is apearing .

Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-07T22:19:43.64+00:00
@Sharukh Kundagol I was able to run the command. It seems like the issue is with the mount point. Can you please run this command to check the mounts to see if it is mounted correctly?

display(dbutils.fs.mounts())

https://docs.databricks.com/storage/azure-storage.html
Sharukh Kundagol 145 Reputation points

2023-04-08T06:08:43.28+00:00
Hi Bhargav, if i run that command in NB it is working, but my issue is something different. I have to run that command as part of inti script. below is my init script. if I create the libraries-init.sh with below command, the script works fine.

dbutils.fs.put("dbfs:/databricks/scripts/libraries-init.sh",""" #!/bin/bash echo "Installing libraries..." pip install -r /dbfs/databricks/scripts/libraries.txt echo "Libraries installed." echo "Capturing library versions..." pip freeze > /dbfs/databricks/scripts/library_versions.txt echo "Library versions captured." """,True)

as i can not run the command in higher environments, i need to deploy the libraries-init.sh file using devops or even copy the file to that location by some method from adls. then same script is not working. for example like below

%sh cp /mnt/adls/someloca/libraries-init.sh /dbfs/databricks/scripts/libraries-init.sh

%sh sh /dbfs/databricks/scripts/libraries-init.sh

throwing error. as mentioned above.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-10T22:41:11.4766667+00:00
It seems like the issue is related to the file permissions. When you use dbutils.fs.put to create the libraries-init.sh file, the file is created with the correct permissions, which allows you to run the script without any issues. However, when you copy the file from ADLS to DBFS using %sh cp command, the file permissions might not be set correctly, which can cause the script to fail.

Can you please try setting the file permissions explicitly after copying the file to DBFS using the following command:

%sh cp /mnt/adls/someloca/libraries-init.sh /dbfs/databricks/scripts/libraries-init.sh chmod +x /dbfs/databricks/scripts/libraries-init.sh

The chmod +x command sets the executable permission on the libraries-init.sh file. I hope this helps resolve the issue.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-12T15:55:10.91+00:00

@Sharukh Kundagol Did you get a chance to look into my earlier response? Please let me know if you have any further questions.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-13T16:58:31.7+00:00

@Sharukh Kundagol I am checking to see if you got a chance to look into my earlier response.
Sharukh Kundagol 145 Reputation points

2023-04-14T18:19:40.17+00:00

HI BhargavaGunnam, I tried Once , but it did not work. i want to give one more try. apologies for delay
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-14T19:04:08.78+00:00

@Sharukh Kundagol Thanks for the reply. Are you getting the same error?
Sharukh Kundagol 145 Reputation points

2023-04-15T19:05:10.1866667+00:00

@Bhargava-MSFT I am getting same error post executable permission.

and also i checked the permission on the file while deployed via CI/CD and dbutils.fs.put, in both case it is has executable permission.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-19T22:32:53.8166667+00:00

@Sharukh Kundagol, To troubleshoot the issue further, I suggest submitting a support ticket so the support engineer can live troubleshoot the issue. In case if you don't have a support plan, please let me know so that I can enable one-time free support ticket for you to work on the issue. I am looking forward to hearing from you.

Your answer

Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-07T22:19:43.64+00:00

@Sharukh Kundagol I was able to run the command. It seems like the issue is with the mount point. Can you please run this command to check the mounts to see if it is mounted correctly?

display(dbutils.fs.mounts())

https://docs.databricks.com/storage/azure-storage.html
Sharukh Kundagol 145 Reputation points

2023-04-08T06:08:43.28+00:00

Hi Bhargav, if i run that command in NB it is working, but my issue is something different. I have to run that command as part of inti script. below is my init script. if I create the libraries-init.sh with below command, the script works fine.

dbutils.fs.put("dbfs:/databricks/scripts/libraries-init.sh",""" #!/bin/bash echo "Installing libraries..." pip install -r /dbfs/databricks/scripts/libraries.txt echo "Libraries installed." echo "Capturing library versions..." pip freeze > /dbfs/databricks/scripts/library_versions.txt echo "Library versions captured." """,True)

as i can not run the command in higher environments, i need to deploy the libraries-init.sh file using devops or even copy the file to that location by some method from adls. then same script is not working. for example like below

%sh cp /mnt/adls/someloca/libraries-init.sh /dbfs/databricks/scripts/libraries-init.sh

%sh sh /dbfs/databricks/scripts/libraries-init.sh

throwing error. as mentioned above.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-10T22:41:11.4766667+00:00

It seems like the issue is related to the file permissions. When you use dbutils.fs.put to create the libraries-init.sh file, the file is created with the correct permissions, which allows you to run the script without any issues. However, when you copy the file from ADLS to DBFS using %sh cp command, the file permissions might not be set correctly, which can cause the script to fail.

Can you please try setting the file permissions explicitly after copying the file to DBFS using the following command:

%sh cp /mnt/adls/someloca/libraries-init.sh /dbfs/databricks/scripts/libraries-init.sh chmod +x /dbfs/databricks/scripts/libraries-init.sh

The chmod +x command sets the executable permission on the libraries-init.sh file. I hope this helps resolve the issue.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-12T15:55:10.91+00:00

@Sharukh Kundagol Did you get a chance to look into my earlier response? Please let me know if you have any further questions.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-13T16:58:31.7+00:00

@Sharukh Kundagol I am checking to see if you got a chance to look into my earlier response.
Sharukh Kundagol 145 Reputation points

2023-04-14T18:19:40.17+00:00

HI BhargavaGunnam, I tried Once , but it did not work. i want to give one more try. apologies for delay
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-14T19:04:08.78+00:00

@Sharukh Kundagol Thanks for the reply. Are you getting the same error?
Sharukh Kundagol 145 Reputation points

2023-04-15T19:05:10.1866667+00:00

@Bhargava-MSFT I am getting same error post executable permission.

and also i checked the permission on the file while deployed via CI/CD and dbutils.fs.put, in both case it is has executable permission.
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2023-04-19T22:32:53.8166667+00:00

@Sharukh Kundagol, To troubleshoot the issue further, I suggest submitting a support ticket so the support engineer can live troubleshoot the issue. In case if you don't have a support plan, please let me know so that I can enable one-time free support ticket for you to work on the issue. I am looking forward to hearing from you.

Share via

init script capture the Libraries

Your answer