.Net UDF for Apache Spark must be callable from Azure Databricks Notebook

Neeraj Kumar Singh 1 Reputation point
2022-10-18T06:31:21.633+00:00

I have a .Net Console Application which perform some operation on given inputs and provide outputs. Have written Spark Wrapper on that, and locally works fine. Facing issue to install this .NET publish packages and dependencies into an Azure Databricks Cluster (with this Notebook is attached).

using Microsoft.Spark.Sql;
using System;
namespace MySparkApp
{
class Program
{
static void Main(string[] args)
{
// Create a Spark session
SparkSession spark = SparkSession
.Builder()
.AppName("word_count_sample")
.GetOrCreate();

         //Register UDFs  
           Func
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,017 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,301 Reputation points Microsoft Employee
    2022-10-19T10:47:32.667+00:00

    Hi @Neeraj Kumar Singh ,

    I could see the steps of deploying your .NET app to databricks workspace. Kindly check this link.

    Did you tried it? Where exactly you stuck? If I am not wrong, you can publish your app to databricks and submit it as job to run. If you are looking to for calling some function which is inside your .NET app form Databricks notebook then thats not possible. Azure databaricks will not support C# languague. You need to consider using Synapse Analytics in that case. Because there we have native support for C# as well.

    Please let me know how it goes.