Azure Synapse .NET C# Sparkpool: Fail to start interpreter

Question

When I am working on a .NET Spark (C#) Notebook in Azure Synapse I always get the following error message.

Fail to start interpreter. detail: org.apache.spark.api.dotnet.DotnetBackend

I have no clue what causes this error message and how to solve it.

I have tried:

• Working in a completely new resource group
• Working with a larger Spark Node
• Running the notebook Hitchhiker's Guide to .NET for Apache Spark available via the Gallery
• Search online for a solution

Answer 1

Hello @Tom Eversdijk

Thanks for the question and using MS Q&A platform.

This error message indicates that there is an issue with the .NET backend for the Apache Spark interpreter in Azure Synapse Analytics. It could be caused by a number of things, such as a missing or incompatible version of the .NET runtime or a problem with the configuration of the interpreter.

You will experience this issue if you are using Azure Synapse Runtime for Apache Spark 3.3

Note: The .NET Core 3.1 library has reached end of support and therefore has been removed from the Azure Synapse Runtime for Apache Spark 3.3, meaning users will no longer be able to access Spark APIs through C# and F# and execute C# code in notebooks and through jobs.

As per the repro here are the results:

Using Azure Synapse Runtime for Apache Spark 3.3

Using Azure Synapse Runtime for Apache Spark 3.2

For more details, refer to Azure Synapse Runtime for Apache Spark 3.3 (Preview)

Hope this helps. Do let us know if you any further queries.

---

Please don’t forget to Accept Answer wherever the information provided helps you, this can be beneficial to other community members.