.NET for Apache Spark: can't read from Event Hubs

Question

.NET for Apache Spark: can't read from Event Hubs

Michael Winick 1

I'm attempting to connect my .NET for Apache Spark application to Event Hubs, and following instructions and code described in:
https://github.com/dotnet/spark/issues/325#issuecomment-552536894

Here's my powershell spark-submit:

spark-submit \--packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.21
--class org.apache.spark.deploy.dotnet.DotnetRunner \--master local
bin\Debug\netcoreapp3.1\microsoft-spark-3-0_2.12-1.0.0.jar `
dotnet bin\Debug\netcoreapp3.1\mySparkApp.dll

with the following output/errors:

130838-errors.txt

Any direction would be greatly appreciated.

Michael Winick 1

        SparkSession spark = SparkSession
            .Builder()
            .AppName("VS Telemetry processing using Spark.NET")
            .GetOrCreate();

        DataFrame events = spark
            .ReadStream()
            .Format("kafka")
            .Options(EventHubConnection.GetEventHubConnectionSettings(10))
            .Load();

        events
            .Select(
                FromJson(Col("body").Cast("string"), "BlobPath String").Alias("Raw"),
                Col("properties"),
                Col("enqueuedTime")
            )
            .SelectExpr("Raw.*", "properties", "enqueuedTime")
            .WithWatermark("enqueuedTime", "60 seconds")
            .CreateOrReplaceTempView("Messages");

StreamingQuery query = spark
.Sql("SELECT * FROM Messages")
.WriteStream()
.OutputMode("append")
.Format("csv")
.Start(@"C:\arcus\mySparkApp\spark_pulled_from_event_hubs.txt");

        Console.WriteLine("Selected from Messages");

        query.AwaitTermination();

Michael Winick 1 Reputation point

2021-09-10T04:07:44.337+00:00

SparkSession spark = SparkSession .Builder() .AppName("VS Telemetry processing using Spark.NET") .GetOrCreate(); DataFrame events = spark .ReadStream() .Format("kafka") .Options(EventHubConnection.GetEventHubConnectionSettings(10)) .Load(); events .Select( FromJson(Col("body").Cast("string"), "BlobPath String").Alias("Raw"), Col("properties"), Col("enqueuedTime") ) .SelectExpr("Raw.*", "properties", "enqueuedTime") .WithWatermark("enqueuedTime", "60 seconds") .CreateOrReplaceTempView("Messages");

StreamingQuery query = spark
.Sql("SELECT * FROM Messages")
.WriteStream()
.OutputMode("append")
.Format("csv")
.Start(@"C:\arcus\mySparkApp\spark_pulled_from_event_hubs.txt");

Console.WriteLine("Selected from Messages"); query.AwaitTermination();

.NET for Apache Spark: can't read from Event Hubs

Activity