Reusing ADLS paths on failed spark streaming.

Peconcillo, Paul Ryan 1

Hi,

We have an Event hub which is used to ingest data from an external sending application. We are reading data from the Eventhub using Azure databricks using Pyspark and stream it on a ADLS sink. But in the event of a failure of the streaming for some reason when I tried to re-run my spark streaming on Azure databricks I got this error "java.lang.NoSuchMethodError: org.apache.spark.sql.eventhubs.EventHubsSource$$anon$1.parseVersion(Ljava/lang/String;I)I" . To run my spark streaming again, I need to delete my sink in the ADLS and restart, its kind of annoying and you should not delete all the events generated after it failed.

HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2022-03-08T17:51:57.537+00:00

Hello @Peconcillo, Paul Ryan ,
Thanks for the ask and using Microsoft Q&A platform.
Can you please sharing the code snippet ? That will help us look into this issue further .
Thanks
Himanshu
Peconcillo, Paul Ryan 1 Reputation point

2022-03-09T10:37:42.077+00:00

iHi Below is my code snippet

Read Event Hub's stream
conf = {}
cnxString = event_hub_endpoint
conf["eventhubs.connectionString"] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(cnxString)

read_from_ehub_df = (
spark.readStream
.format("eventhubs")
.options(**conf)
.load()
)

Decode body column
decoded_from_ehub_df = (
read_from_ehub_df .withColumn("body", read_df["body"].cast("string"))
.select(col("sequenceNumber"),col("body"),col("enqueuedTime"))
)

Stream from EventHun to ADLSg2
stream_query = (
decoded_from_ehub_df.writeStream
.outputMode("append")
.format("parquet")
.option("checkpointLocation",checkpointPath)
.option("path",outputPath)
.start()
)
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2022-03-23T18:32:19.473+00:00

Hello @Peconcillo, Paul Ryan ,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.
Thanks
Himanshu
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2022-03-24T18:47:52.627+00:00

Hello @Peconcillo, Paul Ryan ,
We haven’t received any email from you as requested in the last response and was just checking back to see if you have a resolution . In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, please respond back with the more details as requested and we will try to help .
Thanks
Himanshu