.Net UDF for Apache Spark must be callable from Azure Databricks Notebook

Question

I have a .Net Console Application which perform some operation on given inputs and provide outputs. Have written Spark Wrapper on that, and locally works fine. Facing issue to install this .NET publish packages and dependencies into an Azure Databricks Cluster (with this Notebook is attached).

using Microsoft.Spark.Sql;
using System;
namespace MySparkApp
{
class Program
{
static void Main(string[] args)
{
// Create a Spark session
SparkSession spark = SparkSession
.Builder()
.AppName("word_count_sample")
.GetOrCreate();

         //Register UDFs  
           Func

Answer 1

Hi @Neeraj Kumar Singh ,

I could see the steps of deploying your .NET app to databricks workspace. Kindly check this link.

Did you tried it? Where exactly you stuck? If I am not wrong, you can publish your app to databricks and submit it as job to run. If you are looking to for calling some function which is inside your .NET app form Databricks notebook then thats not possible. Azure databaricks will not support C# languague. You need to consider using Synapse Analytics in that case. Because there we have native support for C# as well.

Please let me know how it goes.