Looking for HDInsight Script Action sample scripts for installing python package in PySpark 3

Alphonse Okossi 141

1 answer

PRADEEPCHEEKATLA-MSFT 77,336 Reputation points Microsoft Employee

2021-03-10T07:18:49.52+00:00
Hello @Alphonse Okossi ,

Steps to install external Python packages using script action in Azure HDInsight:

****Step1:** Create a shell script.**

Line2 - Creating Python virtual environment using condo

Line3 - Seaborn is the package name that you would like to install. #!/usr/bin/env bash
sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/py35new python=3.5 anaconda --yes
sudo /usr/bin/anaconda/envs/py35new/bin/pip install seaborn
****Step2:** Upload your script Action**

Upload your Script Action to a Storage Account that your chosen HDInsights have access to.

Bash script URI https://<my-storage-account>.blob.core.windows.net/<Storage-Container>/script-actions/<SctiptName>

Example:

https://cheprahdistorage.blob.core.windows.net/chepra-2021-03-10t05-18-30-771z/script-actions/seaborn-package.sh

Step3: Create a script action from Azure Portal.

Select the cluster that you want to run the Script action in Azure portal

On the blade under Configuration, you can find Script Actions

Select that and then click Submit New

Select custom

Give a name

Select nodes to install

A few minutes Later, we have successfully installed external Python packages using script actions in Azure HDInsight.

Hope this helps. Do let us know if you any further queries.

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
Alphonse Okossi 141 Reputation points

2021-03-10T20:41:56.14+00:00

I tried that now it says "failed" but debug information is just "Null"

Also noticed this warning in the data lake storage where I stored the sh script: "The file 'script-actions/pipeline-init2.sh' may not render correctly as it contains an unrecognized extension". here is the file:

#!/usr/bin/env bash sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/py35new python=3.5 anaconda --yes sudo /usr/bin/anaconda/bin/conda install -c anaconda mysql-connector-python -n py35new --yes sudo /usr/bin/anaconda/bin/conda install pandas -n py35new --yes sudo /usr/bin/anaconda/bin/conda install panasql -n py35new --yes

PRADEEPCHEEKATLA-MSFT 77,336 Reputation points Microsoft Employee

2021-03-11T03:42:48.357+00:00

Hello @Alphonse Okossi ,

Click on the failed script action and view properties, you will find error log file called as "error_log": "/var/lib/ambari-agent/data/errors-682.txt".

To known the exact reason for the error - you can SSH to the head node of the cluster and run sudo nano /var/lib/ambari-agent/data/output-682.txt.

Hope this helps. Do let us know if you any further queries.

PRADEEPCHEEKATLA-MSFT 77,336 Reputation points Microsoft Employee

2021-03-12T11:27:25.85+00:00

Hello @Alphonse Okossi ,
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Take care & stay safe!

Alphonse Okossi 141 Reputation points

2021-03-12T22:57:13.213+00:00

I followed your instructions above in addition to that in the following:

https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-python-package-installation

but not able to pass the script action successfully. Script runs for hours and ends up failing.

Alphonse Okossi 141 Reputation points

2021-03-12T22:58:40.857+00:00

Is there a runtime way to submit python 3 packages using the spark configration in PySpark Jupyter Notebook?

PRADEEPCHEEKATLA-MSFT 77,336 Reputation points Microsoft Employee

2021-03-15T09:14:15.18+00:00

Hello @Alphonse Okossi ,

Unfortunately, there is no runtime way to submit python 3 packages.

You can try the above method to install the external python packages. Since, the script is failing - I would request you to checkout the error log to know the exact reason.

Hope this helps. Do let us know if you any further queries.

---------------------------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

PRADEEPCHEEKATLA-MSFT 77,336 Reputation points Microsoft Employee

2021-03-16T10:46:47.973+00:00

Hello @Alphonse Okossi ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Sign in to comment