Редагувати

Поділитися через

Azure Synapse Runtime for Apache Spark 3.3 (EOSA)

  • Стаття

Azure Synapse Analytics supports multiple runtimes for Apache Spark. This document covers the runtime components and versions for the Azure Synapse Runtime for Apache Spark 3.3.

Warning

End of support announced for Azure Synapse Runtime for Apache Spark 3.3 July 12th, 2024.

We strongly recommend you upgrade your Apache Spark 3.3 based workloads to Azure Synapse Runtime for Apache Spark 3.4 (GA). For up-to-date information, a detailed list of changes, and specific release notes for Spark runtimes, check and subscribe to Spark Runtimes Releases and Updates.

Component versions

Component Version
Apache Spark 3.3.1
Operating System Ubuntu 18.04
Java 1.8.0_282
Scala 2.12.15
Hadoop 3.3.3
Delta Lake 2.2.0
Python 3.10
R (Preview) 4.2.2

Synapse-Python310-CPU.yml contains the list of libraries shipped in the default Python 3.10 environment in Azure Synapse Spark.

Important

.NET for Apache Spark

  • The .NET for Apache Spark is an open-source project under the .NET Foundation that currently requires the .NET 3.1 library, which has reached the out-of-support status. We would like to inform users of Azure Synapse Spark of the removal of the .NET for Apache Spark library in the Azure Synapse Runtime for Apache Spark version 3.3. Users may refer to the .NET Support Policy for more details on this matter.

  • As a result, it will no longer be possible for users to utilize Apache Spark APIs via C# and F#, or execute C# code in notebooks within Synapse or through Apache Spark Job definitions in Synapse. It is important to note that this change affects only Azure Synapse Runtime for Apache Spark 3.3 and above.

  • We will continue to support .NET for Apache Spark in all previous versions of the Azure Synapse Runtime according to their lifecycle stages. However, we do not have plans to support .NET for Apache Spark in Azure Synapse Runtime for Apache Spark 3.3 and future versions. We recommend that users with existing workloads written in C# or F# migrate to Python or Scala. Users are advised to take note of this information and plan accordingly.

Libraries

To check the libraries included in Azure Synapse Runtime for Apache Spark 3.3 for Java/Scala, Python, and R go to Azure Synapse Runtime for Apache Spark 3.3

Next steps

Migration between Apache Spark versions - support

For guidance on migrating from older runtime versions to Azure Synapse Runtime for Apache Spark 3.3 or 3.4 refer to Runtime for Apache Spark Overview.