can we connect to a eventhub in databricks dynamically if we know the Event hub Namespace and Event hub Name using pyspark?

Shivasai 21 Reputation points
2022-03-04T04:54:55.61+00:00

Requirement: I know the eventhub namespace but we to connect to different event hubs within this EH NS dynamically based on the client requirement (i.e, to which event hub we need to stream data) .

Questions:

is there is a way that we can connect to event hub dynamically if we know the Event hub Namespace and Event hub Name using pyspark?

1) Can we get/access the connection string for the event hub in databricks using python or pyspark. I have see some options on how to get a particular event hub connection string using azure cli but can we get even thub connections strings in databricks using python ?

2) If we know the connection string of Event hub Namespace and we need to connect to an eventhub within this namespace. Can we use EH NS connection string and set some property (i.e, eventhub name) and connect to the eventhub using EH NS connection string in Pyspark ehconf code ?

Note: EH-NS is Eventhub Namsespace

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
568 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,971 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,036 Reputation points
    2022-03-07T21:27:23.597+00:00

    Hello @Shivasai ,
    Thanks for the question and using MS Q&A platform.

    As we understand the ask here is "How can I build the Event Hub connection string in my Databricks notebook, rather than fetching from portal?".

    The good news, is we have a tool for just that, The Connection String Builder. It takes the namespace name, hub name, and SAS key & name. You already mentioned the first two, all you still need is the SAS part.

    Link to Connection String Builder in documentaiton

    import org.apache.spark.eventhubs.ConnectionStringBuilder  
    
    val connectionString = ConnectionStringBuilder()  
      .setNamespaceName("<namespace-name>")  
      .setEventHubName("<eventhub-name>")  
      .setSasKeyName("<key-name>")  
      .setSasKey("<key>")  
      .build  
    

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators