Share via

Spark version control in Azure: integration runtimes vs spark pools

Anonymous
2024-08-15T09:25:00.9466667+00:00

We have received a message that spark 3.3 will reach end of life in 2025, and that we need to upgrade to 3.4. I was able to make new spark pools that use 3.4, but I was unable to do this for the integration runtimes in any way.

Our infrastructure is completely built using data flows and pipelines. From what I understood, these are never run on the spark pools, but only on the integration runtimes. I also understood that the integration runtime spark version control is managed automatically.

Could someone confirm this?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,373 questions
0 comments
Accepted answer
  1. NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff
    2024-08-16T04:08:27.7333333+00:00

    Hi @Dankerlui, Patrick
    Thank you for your question and using the Microsoft Q&A platform.

    Your understanding is correct. In Azure Synapse Analytics, Data Flows and Pipelines do not directly run on the Spark pools that you create. Instead, they operate on integration runtimes, which are managed environments where the execution of these activities takes place.

    The Spark version for integration runtimes is indeed managed automatically by Microsoft. This means that as long as you are using Data Flows or Pipelines, the underlying Spark version is updated by Microsoft without requiring manual intervention. You do not need to manually upgrade Spark versions for integration runtimes as you would for custom Spark pools.

    To summarize:

    • Spark Pools: These are manually managed, and you have to create or update them to use specific Spark versions (like 3.4).
    • Integration Runtimes: The Spark version is automatically managed, and Microsoft will handle updates, including the transition from Spark 3.3 to 3.4.

    For more information, please refer to the below documents:
    https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-version-support
    https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.