@phemanth we have to read data from event-hub by using spark-streaming Like spark.readStream.option
Connect to eventhub using Managed service identity (MSI) from Azure synapse analytics using pyspark spark streaming
Sunil Yadav
0
Reputation points
I am trying to write data to Azure Event Hubs from an Azure Synapse Analytics notebook in PySpark using Managed Service Identity (MSI). Could you please provide step-by-step guidance on how to achieve this, along with the necessary code?
Please note, I have successfully connected using the SAS key previously.
2 answers
Sort by: Most helpful
-
-
phemanth 10,635 Reputation points Microsoft Vendor
2024-09-30T10:34:51.07+00:00 as i mentioned above comment follow the steps adding on top of it
Read Data from Event Hubs
Use Spark Streaming to read data from Event Hubs.
# Define Event Hubs configuration for reading event_hubs_conf = { 'eventhubs.connectionString': connection_string } # Read data from Event Hubs df = spark.readStream \ .format("eventhubs") \ .options(**event_hubs_conf) \ .load() # Process the data (example: print schema) df.printSchema() # Start the streaming query query = df.writeStream \ .outputMode("append") \ .format("console") \ .start() query.awaitTermination() ation()