How to upgrade Spark version in Synaspe?

Pierre Gourven 41

I would like to know how to upgrade Spark version in Synapse?
I have a spark pool on spark 3.1 with a lot of notebooks and would like to upgrade to Spark 3.3 without having to create a new Spark pool and re-deploy my notebooks.
I haven't found a tutorial or explanation on how to achieve this.

Any help would be appreciated

Regards

Pete

BhargavaGunnam-MSFT 25,891 Reputation points Microsoft Employee

2023-02-01T00:37:47.2766667+00:00

Hello @Pierre Gourven,

I am checking to see if you got a chance to look into my above answer.
BhargavaGunnam-MSFT 25,891 Reputation points Microsoft Employee

2023-03-08T17:36:34.5266667+00:00

Hello @Pierre Gourven,

Welcome to the MS Q&A platform.

Sorry, in synapse we can't alter the Apache Spark version, Python, PySpark or Scala/Java/.net.

However, the good news is most of the component versions are the same with spark 3.1 and 3.2

To answer your question:

You will need to add a new spark pool with your desired Apache spark pool version to upgrade your spark pool

I hope this helps. Please let me know if you have any further questions.
BhargavaGunnam-MSFT 25,891 Reputation points Microsoft Employee

2023-03-08T17:57:19.4233333+00:00

Hello @Pierre Gourven,

Thank you for pointing out the document. Earlier, upgrading the version using the PS script didn't work.
But I was able to upgrade it now. It seems like the PS script has been fixed now.

Old execution:

Current execution:
srinivasvarma 0 Reputation points

2023-08-10T10:58:17.9333333+00:00

Can we have a document on how to update this. i.e step by step of process from the beginning.
BhargavaGunnam-MSFT 25,891 Reputation points Microsoft Employee

2023-08-11T16:24:35.7033333+00:00

Hello srinivasvarma,

Please follow the below documentation.

https://learn.microsoft.com/en-us/powershell/module/az.synapse/update-azsynapsesparkpool?view=azps-10.2.0&viewFallbackFrom=azps-9.7.0
Abhijeet Singh 25 Reputation points

2024-01-10T06:27:35.7733333+00:00

Hi @BhargavaGunnam-MSFT I have done the above and it seems to have worked and updated the spark version from 3.1 to 3.3. But now the Spark pool is not working. It give interpreter died error when I run any cell in notebook.

Accepted answer

PRADEEPCHEEKATLA-MSFT 76,746 Reputation points Microsoft Employee

2023-04-28T05:26:25.5433333+00:00

@Pierre Gourven - Yes, you can use Azure Powershell cmdlets to upgrade Apache Spark pool from 3.1 to 3.3.

Update-AzSynapseSparkPool - Updates a Apache Spark pool in Azure Synapse Analytics.

As per repro, I was able to upgrade Apache Spark pool from 3.1 to 3.3 using Update-AzSynapseSparkPool powershell cmdlet as shown below.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.

2 people found this answer helpful.
Imran Akbar 25 Reputation points

2023-11-14T10:58:04.2033333+00:00

Hi @PRADEEPCHEEKATLA-MSFT
I have done the above and it seems to have worked and updated the spark version from 3.1 to 3.3. But now the Spark pool is not working. It give interpreter died error when I run any cell in notebook.

if i look in monitoring section of spark pool, i see the below error:

error code: LIVY_JOB_STATE_ERROR

Message:
[plugins.ukazunpduks324synapse20.DMT.1228 WorkspaceType:<Synapse> CCID:<e1563a8e-b0c4-46d7-a007-2d9746b6759f>] [Monitoring] Livy Endpoint=[https://hubservice2.westeurope.azuresynapse.net:8001/api/v1.0/publish/1c9765de-8d60-4391-bcf5-6d3feec5ea5d]. Livy Id=[2] Job failed during run time with state=[error].

Abhijeet Singh 25 Reputation points

2024-01-10T06:20:17.66+00:00

After updating spark version from 3.1 to 3.3. Notebook fail with error: Interpreter died.How can we resolve this

Sofia Pires 0 Reputation points

2024-01-19T12:43:31.6366667+00:00

I was not able to do that because i do not have any spark pool created in synapse however my dataflows are running in spark pool 3.1. How can I upgraded for 3.3 ? There will be automatic ?
Sign in to comment

1 additional answer

Pierre Gourven 41 Reputation points

2023-03-08T18:00:02.9+00:00

@BhargavaGunnam-MSFT

What about this article?
https://learn.microsoft.com/en-us/powershell/module/az.synapse/update-azsynapsesparkpool?view=azps-9.2.0
Please sign in to rate this answer.

0 comments No comments
Sign in to comment