Reusing ADLS paths on failed spark streaming.

Peconcillo, Paul Ryan 1 Reputation point
2022-03-04T12:15:46.597+00:00

Hi,

We have an Event hub which is used to ingest data from an external sending application. We are reading data from the Eventhub using Azure databricks using Pyspark and stream it on a ADLS sink. But in the event of a failure of the streaming for some reason when I tried to re-run my spark streaming on Azure databricks I got this error "java.lang.NoSuchMethodError: org.apache.spark.sql.eventhubs.EventHubsSource$$anon$1.parseVersion(Ljava/lang/String;I)I" . To run my spark streaming again, I need to delete my sink in the ADLS and restart, its kind of annoying and you should not delete all the events generated after it failed.

.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,438 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
564 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,955 questions
{count} votes