Azure Synapse Runtime for Apache Spark 3.3 (EOSA)
Azure Synapse Analytics supports multiple runtimes for Apache Spark. This document covers the runtime components and versions for the Azure Synapse Runtime for Apache Spark 3.3.
Warning
End of support notification for Azure Synapse Runtime for Apache Spark 3.3
- On July 12th, 2024, end of support was announced for Azure Synapse Runtime for Apache Spark 3.3.
- Effective March 31, 2025, Azure Synapse will discontinue official support for Spark 3.3 Runtimes.
- We strongly recommend you upgrade your Apache Spark 3.3 based workloads to Azure Synapse Runtime for Apache Spark 3.4 (GA).
- For up-to-date information, a detailed list of changes, and specific release notes for Spark runtimes, check and subscribe to Spark Runtimes Releases and Updates.
Component versions
| Component | Version |
|---|---|
| Apache Spark | 3.3.1 |
| Operating System | Ubuntu 18.04 |
| Java | 1.8.0_282 |
| Scala | 2.12.15 |
| Hadoop | 3.3.3 |
| Delta Lake | 2.2.0 |
| Python | 3.10 |
| R (Preview) | 4.2.2 |
Synapse-Python310-CPU.yml contains the list of libraries shipped in the default Python 3.10 environment in Azure Synapse Spark.
Important
.NET for Apache Spark
The .NET for Apache Spark is an open-source project under the .NET Foundation that currently requires the .NET 3.1 library, which has reached the out-of-support status. We would like to inform users of Azure Synapse Spark of the removal of the .NET for Apache Spark library in the Azure Synapse Runtime for Apache Spark version 3.3. Users may refer to the .NET Support Policy for more details on this matter.
As a result, it will no longer be possible for users to utilize Apache Spark APIs via C# and F#, or execute C# code in notebooks within Synapse or through Apache Spark Job definitions in Synapse. It is important to note that this change affects only Azure Synapse Runtime for Apache Spark 3.3 and above.
We will continue to support .NET for Apache Spark in all previous versions of the Azure Synapse Runtime according to their lifecycle stages. However, we do not have plans to support .NET for Apache Spark in Azure Synapse Runtime for Apache Spark 3.3 and future versions. We recommend that users with existing workloads written in C# or F# migrate to Python or Scala. Users are advised to take note of this information and plan accordingly.
Libraries
To check the libraries included in Azure Synapse Runtime for Apache Spark 3.3 for Java/Scala, Python, and R go to Azure Synapse Runtime for Apache Spark 3.3
Next steps
- Manage libraries for Apache Spark pools in Azure Synapse Analytics
- Install workspace packages wheel (Python), jar (Scala/Java), or tar.gz (R)
- Manage packages through Azure PowerShell and REST API
- Manage session-scoped packages
- Apache Spark 3.3.1 Documentation
- Apache Spark Concepts
Migration between Apache Spark versions - support
For guidance on migrating from older runtime versions to Azure Synapse Runtime for Apache Spark 3.3 or 3.4 refer to Runtime for Apache Spark Overview.