Add a package in Synapse Spark pool but failed

Question

Add a package in Synapse Spark pool but failed

Meng Zhang 31 Microsoft Employee

Hi, When I tried to import numba package, it failed with the error:
User's image

I found it is fixed in a new version. https://stackoverflow.com/questions/75094244/shap-systemerror-initialization-of-internal-failed-without-raising-an-except Therefore, I want to update numba. However, it failed no matter I used .txt or .whl.

.whl error: "ERROR: numba-0.56.4-cp310-cp310-win32.whl is not a supported wheel on this platform." .txt error: "ERROR: Cannot uninstall 'llvmlite'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall." And .txt file only has: numba==0.56.4 Can you help me import numba in my spark pool?

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-28T05:59:04.96+00:00

@Meng Zhang - We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

I have provided a detailed answer which has clear steps for which you are looking for. If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-05-03T05:08:13.0466667+00:00

@Meng Zhang - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-28T05:59:04.96+00:00

@Meng Zhang - We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

I have provided a detailed answer which has clear steps for which you are looking for. If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-05-03T05:08:13.0466667+00:00

@Meng Zhang - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

Meng Zhang - Thanks for the question and using MS Q&A platform.

PG Update(26/04/2023 2PM IST): Currently, we are working on fix on improve this error message and the documentation with the changes. The file .tar.gz format is only for R language packages (and not for python). To install python packages please use .yml or .whl file formats.

26/04/2023 11 AM IST:
Looks like there is an issue with install whl file, which would leads to Job failed during run time with state=[dead] File not found. Please validate that file at the provided path exists and that you have the permissions required to access the file.
As per the repro, I had tried to install the numba-0.56.4.tar.gz whl file via workspace packages where I got the same error message as shown above. I will reach out to the internal product team to get more details on why exactly we are experiencing this error message. User's image

Wheanwhile, you can install the numba library by providing a requirements.txt file.
Code snippet in requirement file: numba==0.56.4 User's image

We had successfully installed the numba library using requirements.txt file. User's image

Notebook shows Numba works well on code that looks like this: User's image

Hope this helps. Do let us know if you any further queries.

Meng Zhang 31 Reputation points Microsoft Employee

2023-04-27T08:31:27.8833333+00:00

Hi @PRADEEPCHEEKATLA ,

Thanks for your response. I tried requirements.txt, but it also failed with the error below. Can you help me again?

["isSessionTimedOut":null,"isStreamingQueryExists":"false","impulseErrorCode":"Spark_Ambiguous_NonJvmUserApp_ExitWithStatus1","impulseTsg":null,"impulseClassification":"Ambiguous"},"state":"dead","log":"Pip subprocess error:","ERROR: Cannot uninstall 'llvmlite'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.","","","CondaEnvException: Pip failed",

And this is txt file:

And I add it in this page:
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-27T08:50:27.4366667+00:00

Meng Zhang - I had tested using Apache Spark runtime: 3.3. Could you please share the Spark runtime which you are using?
Meng Zhang 31 Reputation points Microsoft Employee

2023-04-27T09:26:30.9233333+00:00

Do you mean Apache Spark version? It's 3.1
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-27T09:45:07.4066667+00:00

Meng Zhang - Let me test on Apache Spark runtime: 3.1 and get back to you?
Meng Zhang 31 Reputation points Microsoft Employee

2023-04-27T10:03:43.41+00:00

Great. Thanks.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-27T10:16:31.3666667+00:00

Meng Zhang - You are correct when I tried with Apache Spark 3.1 you will experience this error message.

As I suggested above, could you please try using Apache Spark 3.3 version to successfully install the library: numba==0.56.4

In case, if you have any requirement to stick with only Apache Spark 3.1 version, please do let me know.

Hope this helps. Do let us know if you any further queries.
Meng Zhang 31 Reputation points Microsoft Employee

2023-04-28T02:15:07.3766667+00:00

Do I need to create a new Apache Spark pool or I can just upgrade it to 3.3?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-28T05:55:28.3533333+00:00

Meng Zhang - You can use Azure Powershell cmdlets to upgrade Apache Spark pool from 3.1 to 3.3 or you can create a new Spark pool with spark version 3.3.

Update-AzSynapseSparkPool - Updates a Apache Spark pool in Azure Synapse Analytics.

As per repro, I was able to upgrade Apache Spark pool from 3.1 to 3.3 using Update-AzSynapseSparkPool powershell cmdlet as shown below.

After upgrading Apache Spark pool from 3.1 to 3.3, able to successfully install the library: numba==0.56.4

Reason of why you are experiencing - ERROR: Cannot uninstall 'llvmlite'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Note: Apache Spark Pool 3.1 & 3.2 uses - llvmlite=0.36.0 and where as 3.3 uses llvmlite=0.39.1 - where the above issue has been fixed in the latest release.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-05-01T14:41:28.2333333+00:00

Meng Zhang - Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Add a package in Synapse Spark pool but failed

0 additional answers

Your answer