How do I install ydata-profiling on an Azure Synapse spark cluster?

Question

How do I install ydata-profiling on an Azure Synapse spark cluster?

didier caron 20

I am trying to use the ydata-profiling package from pypi and it requires a spark version of 3.3 when I run the %pip install ydata-profiling [notebook,pyspark] I get an interpreter died error. Install steps are described here https://ydata-profiling.ydata.ai/docs/master/pages/getting_started/installation.html

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-02-28T08:50:14.8866667+00:00

@didier caron Could you please share code snippet which you are trying along with the error message which you are experiencing? Or sharing the screenshot will be helpful.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-06T06:48:12.7333333+00:00

@didier caron We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
didier caron 20 Reputation points

2023-03-07T08:20:48.19+00:00

hi. sorry for the late reply. i was out of office last week but here is a screenshot of the error
didier caron 20 Reputation points

2023-03-07T08:23:20.5566667+00:00
running below yields the same result

%pip install ydata-profiling[pyspark]
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-07T09:22:08.8+00:00

@didier caron

You can click on run the notebook to start a new session.

In case, if the new session doesn't work. Kindly share the complete stacktrace of the error message?
didier caron 20 Reputation points

2023-03-07T12:04:34.0566667+00:00

the first cell completed succesfully, the second throws the error and there is no stacktrace to share.. when i run this in a v3.2 spark cluster the installation appears to be working but the package requires v3.3
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-08T08:01:13.22+00:00

@didier caron

Please do check the below answer to resolve the issue.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-09T09:13:41.8966667+00:00

@didier caron Glad to know that the below response was helpful. Thank you for marking this answer as accepted. This will help others find useful answers faster.

Please feel free to take a survey on the relevant answer. You can help us improve by leaving feedback verbatim.

Have a good day!

Regards,
PRADEEPCHEEKATLA-MSFT

Accepted answer

1 additional answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-02-28T08:50:14.8866667+00:00

@didier caron Could you please share code snippet which you are trying along with the error message which you are experiencing? Or sharing the screenshot will be helpful.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-06T06:48:12.7333333+00:00

@didier caron We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
didier caron 20 Reputation points

2023-03-07T08:20:48.19+00:00

hi. sorry for the late reply. i was out of office last week but here is a screenshot of the error
didier caron 20 Reputation points

2023-03-07T08:23:20.5566667+00:00

running below yields the same result

%pip install ydata-profiling[pyspark]
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-07T09:22:08.8+00:00

@didier caron

You can click on run the notebook to start a new session.

In case, if the new session doesn't work. Kindly share the complete stacktrace of the error message?
didier caron 20 Reputation points

2023-03-07T12:04:34.0566667+00:00

the first cell completed succesfully, the second throws the error and there is no stacktrace to share.. when i run this in a v3.2 spark cluster the installation appears to be working but the package requires v3.3
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-08T08:01:13.22+00:00

@didier caron

Please do check the below answer to resolve the issue.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-03-09T09:13:41.8966667+00:00

@didier caron Glad to know that the below response was helpful. Thank you for marking this answer as accepted. This will help others find useful answers faster.

Please feel free to take a survey on the relevant answer. You can help us improve by leaving feedback verbatim.

Have a good day!

Regards,
PRADEEPCHEEKATLA-MSFT

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

@didier caron

Thanks for the question and using MS Q&A platform.

As per the repro, I was experience the same issue when I used %pip and able to resolve the issue by using just pip

User's image

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Hi

Thanks for reaching out to Microsoft Q&A.

Pls go through below links that are close to what you have asked for. I think one of this would def help.

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-manage-pool-packages

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries

https://techcommunity.microsoft.com/t5/analytics-on-azure-blog/install-third-party-whl-packages-within-apache-spark-cluster/ba-p/3667135

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.

Share via

How do I install ydata-profiling on an Azure Synapse spark cluster?

1 additional answer

Your answer