Can I specify command line arguments when deploying .NET for Apache Spark with Spark Submit in Azure Databricks?

Question

I've modified "Get started in 10 minutes" to allow you to specify a file name.

spark-submit ^

--class org.apache.spark.deploy.dotnet.DotnetRunner ^
--master local ^
microsoft-spark-2-4_2.11-1.0.0.jar ^
dotnet MySparkApp.dll input.txt

How do I get it to work this with Azure Databricks?

["--class","org.apache.spark.deploy.dotnet.DotnetRunner","/dbfs/spark-dotnet/microsoft-spark-2-4_2.11-1.0.0.jar","/dbfs/spark-dotnet/publish.zip","MySparkApp"]

Answer 1

Hello @大介 西野 ,

Thanks for the question and using the MS Q&A platform.

You can run your .NET for Apache Spark jobs on Databricks clusters, but it is not available out-of-the-box. There are two ways to deploy your .NET for Apache Spark job to Databricks: spark-submit and Set Jar.

For more details, refer to Submit a .NET for Apache Spark job to Databricks.

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.