EventHub, PySpark : Unable to load messages from Event Hub instance using spark.read.format.options.load() method

Question

EventHub, PySpark : Unable to load messages from Event Hub instance using spark.read.format.options.load() method

I'm using the following Pyspark code:

myConnectionString = "My connection string"
ehConf = {}
ehConf['eventhubs.connectionString'] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(myConnectionString)

df = spark \
.read \
.format("eventhubs") \
.options(**ehConf) \
.load()
df = df.withColumn("body", df["body"].cast("string"))

display(df)

I'm getting a "No data available" message on execution of the above PySpark code.

Answer 1

Hello @K K ,

Welcome to the MS Q&A platform.

Are you creating an Event Hubs Source for Streaming queries or batch queries.

You need to use the Structured Batch read API to read the events from the Event Hub as shown in the following code snippet.

 # Read events from the Event Hub  
 df = spark.read.format("eventhubs").options(**ehConf).load()   
 # Visualize the Dataframe in realtime  
 display(df)  

You need to use the Structured Streaming readStream API to read the events from the Event Hub as shown in the following code snippet.

 # Read events from the Event Hub  
    df = spark.readStream.format("eventhubs").options(**ehConf).load()  
    # Visualize the Dataframe in realtime  
    display(df)  

For more details, refer to Structured Streaming + Event Hubs Integration Guide for PySpark.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators