Connect to eventhub using Managed service identity (MSI) from Azure synapse analytics using pyspark spark streaming

Sunil Yadav 0 Reputation points
2024-09-26T05:45:48.2766667+00:00

I am trying to write data to Azure Event Hubs from an Azure Synapse Analytics notebook in PySpark using Managed Service Identity (MSI). Could you please provide step-by-step guidance on how to achieve this, along with the necessary code?

Please note, I have successfully connected using the SAS key previously.

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
641 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,927 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Sunil Yadav 0 Reputation points
    2024-09-27T14:04:08.7333333+00:00

    @phemanth we have to read data from event-hub by using spark-streaming Like spark.readStream.option

    0 comments No comments

  2. phemanth 10,635 Reputation points Microsoft Vendor
    2024-09-30T10:34:51.07+00:00

    @Sunil Yadav

    as i mentioned above comment follow the steps adding on top of it

    Read Data from Event Hubs

    Use Spark Streaming to read data from Event Hubs.

    # Define Event Hubs configuration for reading
    event_hubs_conf = {
        'eventhubs.connectionString': connection_string
    }
    # Read data from Event Hubs
    df = spark.readStream \
        .format("eventhubs") \
        .options(**event_hubs_conf) \
        .load()
    # Process the data (example: print schema)
    df.printSchema()
    # Start the streaming query
    query = df.writeStream \
        .outputMode("append") \
        .format("console") \
        .start()
    query.awaitTermination()
    ation()
    
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.