Retirement notice: Transition to Azure Synapse runtime for Apache Spark 3.2 or 3.3

PP007 20 Reputation points
2023-07-26T07:37:05.7533333+00:00

Hello,

We got "Retirement notice: Transition to Azure Synapse runtime for Apache Spark 3.2 or 3.3" notice in last month. We are currently using data flows and a dedicated SQL pool in Synapse. As far as I know, there is no option for selecting a spark version when creating a data flow or dedicated SQL pool. Do we need to take any action on this or will the pipelines/data flows automatically adapt to the Spark version change? Our dataflow workloads uses Auto Resolve IR.

Thanks in advance for any help you can provide.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,186 questions
0 comments No comments
{count} votes

Accepted answer
  1. Bhargava-MSFT 31,206 Reputation points Microsoft Employee
    2023-07-31T17:03:09.6666667+00:00

    Hello PP007,

    You don't need to take any action if you are using the dataflows and your understanding is correct.

    The notification you have received only applies to "Apache Spark pools in the synapse."

    The pipelines/data flows will automatically adapt to the Spark version change without any action required from your end.

    For Apache Spark Pools:

    To update the runtime in the spark pool, you can use the PS command: Update-AzSynapseSparkPool

    https://learn.microsoft.com/en-us/powershell/module/az.synapse/update-azsynapsesparkpool?view=azps-10.1.0&viewFallbackFrom=azps-9.2.0

    From the document:

    On 29 September 2023, Azure Synapse runtime for Apache Spark 2.4 will be retired in accordance with the Synapse runtime for Apache Spark lifecycle policy, and any workloads still using it will stop running. Before that date, you'll need to transition your workloads to version 3.2 or 3.3. 

     User's image

    I hope this helps.

    If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. JimmySalian-2011 42,241 Reputation points
    2023-07-26T10:20:55.93+00:00

    Hi,

    Yes you will need to take action to migrate to the new version and there is a detailed blog on this and also migration guide if it is a legacy unsupported code.

    On 29 September 2023, Azure Synapse runtime for Apache Spark 2.4 will be retired in accordance with the Synapse runtime for Apache Spark lifecycle policy, and any workloads still using it will stop running. Before that date, you'll need to transition your workloads to version 3.2 or 3.3. 

     https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/azure-synapse-analytics-february-update-2023/ba-p/3758812#:~:text=On%2029%20September%202023%2C%20Azure,to%20version%203.2%20or%203.3.

    Hope this helps.

    JS

    ==

    Please Accept the answer if the information helped you. This will help us and others in the community as well.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.