Having issue installing mysql-connector-python package to Jupiter notebook on PySpark3 using Script Action-script failing

Question

Having issue installing mysql-connector-python package to Jupiter notebook on PySpark3 using Script Action-script failing

Alphonse Okossi 141

Here is my Script Action commands:

!/usr/bin/env bash

sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/bgdatamysql python=3.5 anaconda --yes
sudo /usr/bin/anaconda/bin/conda install -c anaconda mysql-connector-python -n bgdatamysql --yes
sudo /usr/bin/anaconda/bin/conda install pandas -n bgdatamysql --yes
sudo /usr/bin/anaconda/bin/conda install panasql -n bgdatamysql --yes

Script action failing

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-09T04:54:19.953+00:00
Hello @Alphonse Okossi ,

Thanks for the ask and using the Microsoft Q&A platform.

In order to understand your issue - could you share the below details:

Could you please share the exact steps on where you are running the script action (Azure HDInsight cluster or Local machine) and how you are trying to execute script action?

When you say "Script action failing", are you seeing any error message?

In case if you find any error message, kindly share the complete stack trace of the error message which you are experiencing?
Alphonse Okossi 141 Reputation points

2021-03-09T14:00:03.167+00:00
I am running the script in HDInsight/Script Action:

I was trying to create a new environment based on so the script I am trying to run is:

!/usr/bin/env bash

sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/bgdatamysql python=3.5 anaconda --yes
sudo /usr/bin/anaconda/bin/conda install -c anaconda mysql-connector-python -n bgdatamysql --yes
sudo /usr/bin/anaconda/bin/conda install pandas -n bgdatamysql --yes
sudo /usr/bin/anaconda/bin/conda install panasql -n bgdatamysql --yes

error:
Status
Failed

I was trying the following https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-python-package-installation

As of the errors not sure if it content some sensitive info
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-10T06:36:20.62+00:00

Hello @Alphonse Okossi ,

Thanks for the details.

To known the exact reason for the error - you can SSH to the head node of the cluster and run sudo nano /var/lib/ambari-agent/data/output-682.txt.

1 answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-09T04:54:19.953+00:00

Hello @Alphonse Okossi ,

Thanks for the ask and using the Microsoft Q&A platform.

In order to understand your issue - could you share the below details:

Could you please share the exact steps on where you are running the script action (Azure HDInsight cluster or Local machine) and how you are trying to execute script action?

When you say "Script action failing", are you seeing any error message?

In case if you find any error message, kindly share the complete stack trace of the error message which you are experiencing?
Alphonse Okossi 141 Reputation points

2021-03-09T14:00:03.167+00:00

I am running the script in HDInsight/Script Action:

I was trying to create a new environment based on so the script I am trying to run is:

!/usr/bin/env bash

sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/bgdatamysql python=3.5 anaconda --yes
sudo /usr/bin/anaconda/bin/conda install -c anaconda mysql-connector-python -n bgdatamysql --yes
sudo /usr/bin/anaconda/bin/conda install pandas -n bgdatamysql --yes
sudo /usr/bin/anaconda/bin/conda install panasql -n bgdatamysql --yes

error:
Status
Failed

I was trying the following https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-python-package-installation

As of the errors not sure if it content some sensitive info
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-10T06:36:20.62+00:00

Hello @Alphonse Okossi ,

Thanks for the details.

To known the exact reason for the error - you can SSH to the head node of the cluster and run sudo nano /var/lib/ambari-agent/data/output-682.txt.

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

Hello @Alphonse Okossi ,

Steps to install external Python packages using script action in Azure HDInsight:

****Step1:** Create a shell script.**

Line2 - Creating Python virtual environment using condo
Line3 - Seaborn is the package name that you would like to install. #!/usr/bin/env bash
sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/py35new python=3.5 anaconda --yes
sudo /usr/bin/anaconda/envs/py35new/bin/pip install seaborn
****Step2:** Upload your script Action**

Upload your Script Action to a Storage Account that your chosen HDInsights have access to.

Bash script URI https://<my-storage-account>.blob.core.windows.net/<Storage-Container>/script-actions/<SctiptName>

Example:

https://cheprahdistorage.blob.core.windows.net/chepra-2021-03-10t05-18-30-771z/script-actions/seaborn-package.sh

Step3: Create a script action from Azure Portal.

Select the cluster that you want to run the Script action in Azure portal
On the blade under Configuration, you can find Script Actions
Select that and then click Submit New
Select custom
Give a name
Select nodes to install

A few minutes Later, we have successfully installed external Python packages using script actions in Azure HDInsight.

Hope this helps. Do let us know if you any further queries.

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-11T12:31:09.69+00:00

Hello @Alphonse Okossi ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-12T11:24:48.67+00:00

Hello @Alphonse Okossi ,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Take care & stay safe!
Raghu Ram Mahankali (Quadrant Resource) 1 Reputation point

2021-04-12T11:57:16.007+00:00

Hi @PRADEEPCHEEKATLA ,

I followed the exact steps and here is the error message I see in head node.
I did sudo nano /var/lib/ambari-agent/data/output-186.txt to see the log.

Fetching package metadata ...........
Solving package specifications:
('Start downloading script locally: ', u'https://accountforhdinsigt.blob.core.windows.net/containerforscripts/create_ne$Fromdos line ending conversion successful
('Unexpected error:', "('Execution of custom script failed with exit code', 1)")
Removing temp location of the script

Share via

Having issue installing mysql-connector-python package to Jupiter notebook on PySpark3 using Script Action-script failing

!/usr/bin/env bash

!/usr/bin/env bash

1 answer

Your answer