Copy Azure Service Bus messages in Queue to Data Lake via Databricks

Question

Copy Azure Service Bus messages in Queue to Data Lake via Databricks

Carlton Patterson 601

I am trying to connect Azure Service Bus to Databricks and send messages in Azure Service Bus to data lake.

I have successfully connected Databricks to Service Bus via ConnectionString, but I'm not sure how to move messages in Service Queue to Lake via Databricks.

Can someone provide some guidance

MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee

2022-02-09T17:26:35.673+00:00

Hello @Carlton Patterson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee

2022-02-11T17:33:15.823+00:00

Hello @@Carlton Patterson ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you .
In case if you have any other question on a different issue, we request you to open a new thread .

Thanks

MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee

2022-02-09T17:26:35.673+00:00

Hello @Carlton Patterson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee

2022-02-11T17:33:15.823+00:00

Hello @@Carlton Patterson ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you .
In case if you have any other question on a different issue, we request you to open a new thread .

Thanks

Answer 1

MartinJaffer-MSFT 25,556 Microsoft Employee

Hello and welcome to Microsoft Q&A @Carlton Patterson .

It is my understanding you want to use Databricks to receive messages from Azure Service Bus queue and write them to some form of storage (Azure Blob storage / Azure Data Lake Gen2). YOu have your connection string, but are uncertain of next steps.

If you are using a Python / PySpark notebook, then you might want to look into the Python example script for receiving a message from a queue, and iterating over a queue. I found the examples here. There are also Java versions.

There are directions for connecting to Data Lake Gen 2 here.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Carlton Patterson 601 Reputation points

2022-02-12T15:04:26.793+00:00

Hi MartinJaffer,

Thanks for reaching out.

I have used the link you suggested. However, the solution doesnt' show you have to write the data to Azure Blob or ADLS.

Would you agree?

MartinJaffer-MSFT 25,556 Microsoft Employee

@ CarltonPatterson-0401

For Azure Blob, you can mount the drive with this code:

dbutils.fs.mount(  
  source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",  
  mount_point = "/mnt/<mount-name>",  
  extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

For Data Lake Gen 1 you can mount or use service principal

val configs = Map(  
  "<prefix>.oauth2.access.token.provider.type" -> "ClientCredential",  
  "<prefix>.oauth2.client.id" -> "<application-id>",  
  "<prefix>.oauth2.credential" -> dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-service-credential>"),  
  "<prefix>.oauth2.refresh.url" -> "https://login.microsoftonline.com/<directory-id>/oauth2/token")  
  
// Optionally, you can add <directory-name> to the source URI of your mount point.  
dbutils.fs.mount(  
  source = "adl://<storage-resource>.azuredatalakestore.net/<directory-name>",  
  mountPoint = "/mnt/<mount-name>",  
  extraConfigs = configs)

Copy Azure Service Bus messages in Queue to Data Lake via Databricks

1 answer

Activity