can we connect to a eventhub in databricks dynamically if we know the Event hub Namespace and Event hub Name using pyspark?

Question

Requirement: I know the eventhub namespace but we to connect to different event hubs within this EH NS dynamically based on the client requirement (i.e, to which event hub we need to stream data) .

Questions:

is there is a way that we can connect to event hub dynamically if we know the Event hub Namespace and Event hub Name using pyspark?

1) Can we get/access the connection string for the event hub in databricks using python or pyspark. I have see some options on how to get a particular event hub connection string using azure cli but can we get even thub connections strings in databricks using python ?

2) If we know the connection string of Event hub Namespace and we need to connect to an eventhub within this namespace. Can we use EH NS connection string and set some property (i.e, eventhub name) and connect to the eventhub using EH NS connection string in Pyspark ehconf code ?

Note: EH-NS is Eventhub Namsespace

Answer 1

Hello @Shivasai ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is "How can I build the Event Hub connection string in my Databricks notebook, rather than fetching from portal?".

The good news, is we have a tool for just that, The Connection String Builder. It takes the namespace name, hub name, and SAS key & name. You already mentioned the first two, all you still need is the SAS part.

Link to Connection String Builder in documentaiton

import org.apache.spark.eventhubs.ConnectionStringBuilder  

val connectionString = ConnectionStringBuilder()  
  .setNamespaceName("<namespace-name>")  
  .setEventHubName("<eventhub-name>")  
  .setSasKeyName("<key-name>")  
  .setSasKey("<key>")  
  .build  

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators