Spark Runtime Support

Question

As per the https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-version-support

The Current GA Version of Spark Runtime is 3.3, with an End-of-Life announcement data expected on Nov 17 2023. Is there a plan for Synapse Runtime 3.4 support? What libraries are included in this? How can we subscribe to a preview version?

Answer 1

@Dinesh Balachandran & Sarah Moens

The end of life date for Spark 3.3 will be extended, as we want to honor the rule - we support runtime for two full years after it has been announced as GA. So please expect the end of life date for spark 3.3 around February 2025. 

In 2024, we plan to fully deprecate runtimes based on Spark 3.1 and Spark 3.2.

Spark 3.4.1 in Synapse is available (in synapse studio) behind a feature flag (https://ms.web.azuresynapse.net/en/?showSpark34=true). This flag will be removed in the upcoming days, with further announcements.

Note: Spark 3.4 will be in Public Preview stage in Synapse, and GA is planned for 2024. We do not have yet the full list of libraries. But, you can check Java/Scala and R libraries (these libraries will remain the same) here https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-1-2. Python packages will be updated during public preview stage.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

@Dinesh Balachandran - Thanks for the question and using MS Q&A platform.

According to the official documentation, Azure Synapse Analytics currently supports Spark Runtime version 3.3. However, Microsoft is constantly working to improve the service and add new features.

Regarding the support for Spark Runtime version 3.4, our product team is currently working and I do not have any information on the release date. Microsoft usually announces new features and updates through official channels such as the Azure blog or the Azure updates page. I recommend keeping an eye on those channels for any announcements related to Spark Runtime version 3.4 support. And also, I will update the answer once it's available.

As for the libraries included in Spark Runtime version 3.4, I do not have that information at the moment. However, you can refer to the official Apache Spark documentation for a list of libraries included in the latest version of Spark.

The following chart captures a typical lifecycle path for a Synapse runtime for Apache Spark.

For the complete runtime for Apache Spark lifecycle and support policies, refer to Synapse runtime for Apache Spark lifecycle and supportability.

Regarding the preview version, you can sign up for the Azure Synapse Analytics preview program to get early access to new features and updates. For specific feature product team will ask you sign-up for the feature via MS Forms to get access to the preview features - which purely depends on product team decision on giving access to you or not.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.