EventHub, PySpark : Unable to load messages from Event Hub instance using spark.read.format.options.load() method

K K 1 Reputation point
2022-04-01T20:41:48.573+00:00

EventHub, PySpark : Unable to load messages from Event Hub instance using spark.read.format.options.load() method

I'm using the following Pyspark code:

myConnectionString = "My connection string"
ehConf = {}
ehConf['eventhubs.connectionString'] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(myConnectionString)

df = spark \
.read \
.format("eventhubs") \
.options(**ehConf) \
.load()
df = df.withColumn("body", df["body"].cast("string"))

display(df)

I'm getting a "No data available" message on execution of the above PySpark code.

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
580 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 82,196 Reputation points Microsoft Employee
    2022-04-04T08:49:14.44+00:00

    Hello @K K ,

    Welcome to the MS Q&A platform.

    Are you creating an Event Hubs Source for Streaming queries or batch queries.

    You need to use the Structured Batch read API to read the events from the Event Hub as shown in the following code snippet.

     # Read events from the Event Hub  
     df = spark.read.format("eventhubs").options(**ehConf).load()   
     # Visualize the Dataframe in realtime  
     display(df)  
    

    You need to use the Structured Streaming readStream API to read the events from the Event Hub as shown in the following code snippet.

     # Read events from the Event Hub  
        df = spark.readStream.format("eventhubs").options(**ehConf).load()  
        # Visualize the Dataframe in realtime  
        display(df)  
    

    For more details, refer to Structured Streaming + Event Hubs Integration Guide for PySpark.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators