Interpreter died error after updating spark pool from 3.1 to 3.3 in Azure synapse

Abhijeet Singh 25

I have updated the spark version from 3.1 to 3.3 in spark pool . But now the Spark pool is not working. It give interpreter died error when I run any cell in notebook.

Abhijeet Singh 25 Reputation points

2024-01-10T06:41:10.6366667+00:00

I have used below method to update
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-01-10T19:57:56.97+00:00

Hello Abhijeet Singh, Can you please let me know if this is intermittent or continuously failing after the upgrade?
Abhijeet Singh 25 Reputation points

2024-01-10T20:00:50.67+00:00

Continuously failing after update
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-01-10T20:30:28.0266667+00:00

Abhijeet Singh, Thank you for the confirmation. I am reaching out to my internal team and get back to you as soon as I hear from them.
Alex Menezes dos Santos 0 Reputation points

2024-01-19T17:03:29.7966667+00:00

I'm having the same error, how to solve it?
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-01-19T17:12:18.1733333+00:00

Hello Alex Menezes dos Santos and Abhijeet Singh, Product group is working on the fix. I will update you as soon as I hear from them. Thank you for your patience.
Edwin van Halem 0 Reputation points

2024-01-29T13:53:56.3033333+00:00

We are currently experiencing exactly the same problem. On our other environment, where we did not update the spark version, everything is working just fine. It's clear to me that that update is a direct cause of the problem.
i am not able to run any notebook since the update.
Xiaomei Ren 0 Reputation points Microsoft Employee

2024-02-23T12:33:22.01+00:00

Having the same issue, any updates?
Pavlina 0 Reputation points

2024-04-24T14:49:09.4633333+00:00

In our case the issue was caused by custom libraries attached to the pools. For a workaround in that case, you can: Downgrade back -> remove attached libraries -> update Spark version -> re-attach libraries.

Or create a brand new pool, if that is an option.

(Same is explained in the migration documentation, but not very clear)

1 answer

Edwin van Halem 0 Reputation points

2024-02-09T08:59:43.6666667+00:00

What is the resolution to this issue?
I used the same Azure CLI update command. Any Python notebook instantly fails after spark session is ready. Interpreter died:
error message: on our other environment, where we did not do the update yet, there is no problem running python. The update command is the direct cause of this problem.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment