How to upgrade the Azure Synapse Runtime for Apache Spark 3.1 to version 3.3

Upul Karunaratne 20 Reputation points
2024-01-12T10:45:46.7933333+00:00

We received an email regarding the retirement of Apache Spark 3.1 as of January 26, 2024.

There is a notice saying to upgrade to Apache Spark 3.3 when clicking on [Azure Synapse Link for Dataverse] from Power Apps, but I am not seeing the Update option on the Azure Synapse Link for Dataverse. We don't have any defined Apache Spark Pools in Synapse, but it's running Apache Spark Application batch jobs, which are Synapse Pipelines triggered daily. Integration runtime dataflows listed under the Apache Spark Application section are using Apache Spark Version 3.1; however, when checked in Synapse using Powershell Get-AzSynapseSparkPool, it is showing the Spark version as 3.3. Questions

  1. How to upgrade the Azure Synapse Runtime to version 3.3 before it gets automatically upgraded on January 26.
  2. Do we need to upgrade as we don't have any defined Spark pools?
  3. Will the existing pipelines and dataflows work or be lost when Microsoft upgrades the Spark version to 3.3?
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,997 questions
0 comments No comments
{count} votes

Accepted answer
  1. Smaran Thoomu 16,890 Reputation points Microsoft Vendor
    2024-01-12T12:57:49.73+00:00

    Hi @Upul Karunaratne , Thank you for posting query on Microsoft Q&A Platform.

    Yes, you can use Azure Powershell cmdlets to upgrade Apache Spark pool from 3.1 to 3.3.

    Update-AzSynapseSparkPool - Updates a Apache Spark pool in Azure Synapse Analytics.

    As per repro, I was able to upgrade Apache Spark pool from 3.1 to 3.3 using Update-AzSynapseSparkPool powershell cmdlet as shown below. User's image Regarding your second question, if you don't have any defined Spark pools, you don't need to upgrade them. However, if you are running Apache Spark Application batch jobs, which are Synapse Pipelines triggered daily, you should upgrade the Spark version to 3.3 to avoid any issues.

    Regarding your third question, existing pipelines and dataflows should work after upgrading to Spark version 3.3. However, it's always a good practice to test your pipelines and dataflows after upgrading to a new version of Spark to ensure everything is working as expected.

    Finally, regarding the automatic upgrade of Spark version to 3.3 on January 26, 2024, Microsoft will handle the upgrade process, and you don't need to do anything. However, it's always a good practice to upgrade to the latest version of Spark to take advantage of new features and bug fixes.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.