When will Apache Spark 3.4 Azure be available for Synapse Runtime?

Question

When will Apache Spark 3.4 Azure be available for Synapse Runtime?

Amit Amola 25 Microsoft Employee

Our Data Science team at Identity Microsoft Dublin, Ireland, primarily uses Azure Synapse. With support for Apache Spark 3.3 ending in March, we need to migrate our jobs to version 3.4. Previously, we upgraded directly from 3.1 to 3.3 as it was available at that time. Could you provide the timeline for Spark 3.5 support in Synapse Runtime? If it becomes available before the 3.4 migration deadline, it would be beneficial for us to upgrade directly to 3.5.

Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T17:14:36.29+00:00

For anyone else looking for same answer:

Apache Spark 3.5 is currently in the evaluation phase, and they do not have a confirmed release timeline for Synapse Runtime. Based on the current roadmap, it is unlikely that Spark 3.5 will be available before the retirement date of Spark 3.3 (March 31, 2025).

They recommend migrating the workloads to Spark 3.4 to ensure continued support and to take advantage of the improvements it offers.

Accepted answer

1 additional answer

Your answer

Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T17:14:36.29+00:00

For anyone else looking for same answer:

Apache Spark 3.5 is currently in the evaluation phase, and they do not have a confirmed release timeline for Synapse Runtime. Based on the current roadmap, it is unlikely that Spark 3.5 will be available before the retirement date of Spark 3.3 (March 31, 2025).

They recommend migrating the workloads to Spark 3.4 to ensure continued support and to take advantage of the improvements it offers.

Answer 1

Smaran Thoomu 21,620 Microsoft External Staff

Hi @Amit Amola
Welcome to Microsoft Q&A platform.

In addition to the @Shikha Ghildiyal , thank you for reaching out and sharing your use case! We understand the importance of having the latest Apache Spark versions available in Azure Synapse, especially with the upcoming end of support for Apache Spark 3.3.

Currently, Apache Spark 3.4 for Synapse Runtime is under active development, and we are targeting its availability before the March 2025 deadline. We recommend monitoring the Azure Updates page and the Synapse Analytics documentation for the latest announcements regarding its release.

Meanwhile, we suggest planning your migration to Apache Spark 3.4 once it becomes available to ensure your jobs remain supported and leverage the latest enhancements.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T13:25:00.1766667+00:00

"Currently, Apache Spark 3.4 for Synapse Runtime is under active development"

Do you mean Apache Spark 3.5?
Smaran Thoomu 21,620 Reputation points Microsoft External Staff

2025-01-13T13:37:41.26+00:00

Amit Amola Thank you for pointing that out! To clarify:

Currently, Apache Spark 3.4 for Synapse Runtime is under active development, and we are working towards making it available before the March 2025 deadline for Spark 3.3 end-of-support.

Regarding Apache Spark 3.5, it is still in the evaluation stage for Synapse Runtime, and we currently do not have a confirmed timeline for its release. If it becomes available before your migration deadline, we will announce it on the Azure Updates page

I hope this clears up any confusion! Please let us know if you have additional questions.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T13:42:53.8666667+00:00

But Spark 3.4 is already available and has been for quite a while now.

Have been receiving this email as well for last few months:

So it is already available. Not sure I get your point here.

But in any case, Spark 3.5 won't be before the disabling date of 3.3 I guess.
Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T14:03:29.96+00:00

It's been available since April 2024

https://azure.microsoft.com/en-us/updates/?id=general-available-azure-synapse-runtime-for-apache-spark-34-is-now-ga
Smaran Thoomu 21,620 Reputation points Microsoft External Staff

2025-01-13T14:43:20.08+00:00

Amit Amola Thank you for pointing this out, and I appreciate your patience. Let me clarify the situation:

You are correct that Apache Spark 3.4 for Synapse Runtime is already generally available (GA) as of April 2024. My earlier statement about Spark 3.4 being "under development" was incorrect, and I sincerely apologize for the confusion caused.

To address your question about Apache Spark 3.5, it is currently in the evaluation phase, and we do not have a confirmed release timeline for Synapse Runtime. Based on the current roadmap, it is unlikely that Spark 3.5 will be available before the retirement date of Spark 3.3 (March 31, 2025).

We recommend migrating your workloads to Spark 3.4 to ensure continued support and to take advantage of the improvements it offers.

Thank you for bringing this to our attention, and I hope this clears up any confusion.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T17:15:57.98+00:00

Thanks Smaran

I am not able to accept this as answer, as it is a comment.
Smaran Thoomu 21,620 Reputation points Microsoft External Staff

2025-01-13T17:22:19.2633333+00:00

Amit Amola To accept my main answer, please scroll up to the original response (above this comment thread). You should see an “Accept Answer” button below the main answer.

The main response already consolidates all the clarifications and information from this discussion, so accepting it will help others who may have a similar query.

Answer 2

Hi Amit Amola,

Thanks for reaching out to Microsoft Q&A.

Latest version is 3.4 for now.

The following table lists the runtime name, Apache Spark version, and release date for supported Azure Synapse Runtime releases.

Runtime name	Release date	Release stage	End of Support announcement date	End of Support effective date
Azure Synapse Runtime for Apache Spark 3.4	Nov 21, 2023	GA (as of Apr 8, 2024)	Q2 2025	Q1 2026
Azure Synapse Runtime for Apache Spark 3.4	Nov 21, 2023	GA (as of Apr 8, 2024)	Q2 2025	Q1 2026
Azure Synapse Runtime for Apache Spark 3.3	Nov 17, 2022	end of support announced	July 12th, 2024	3/31/2025
Azure Synapse Runtime for Apache Spark 3.2	July 8, 2022	deprecated and soon disabled	July 8, 2023	July 8, 2024
Azure Synapse Runtime for Apache Spark 3.1	May 26, 2021	deprecated and soon disabled	January 26, 2023	January 26, 2024
Azure Synapse Runtime for Apache Spark 2.4	December 15, 2020	deprecated and soon disabled	July 29, 2022	September 29, 2023	For up-to-date information, a detailed list of changes, and specific release notes for Spark runtimes, check and subscribe Spark Runtimes Releases and Updates.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Amit Amola 25 Reputation points Microsoft Employee

2025-01-13T11:59:53.6233333+00:00

This doesn't answer my question about Spark 3.5 support in Synapse at all.
Shikha Ghildiyal 4,545 Reputation points Microsoft Employee

2025-01-13T12:07:25.0333333+00:00

Spark 3.5 support in Synapse is not released yet. Please keep an eye on official page for release notes here- https://github.com/microsoft/synapse-spark-runtime/tree/main/Synapse/spark3.4

Share via

When will Apache Spark 3.4 Azure be available for Synapse Runtime?

1 additional answer

Your answer