Copy Azure Service Bus messages in Queue to Data Lake via Databricks

Carlton Patterson 741

I am trying to connect Azure Service Bus to Databricks and send messages in Azure Service Bus to data lake.

I have successfully connected Databricks to Service Bus via ConnectionString, but I'm not sure how to move messages in Service Queue to Lake via Databricks.

Can someone provide some guidance

MartinJaffer-MSFT 26,011 Reputation points

2022-02-09T17:26:35.673+00:00

Hello @Carlton Patterson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
MartinJaffer-MSFT 26,011 Reputation points

2022-02-11T17:33:15.823+00:00

Hello @@Carlton Patterson ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you .
In case if you have any other question on a different issue, we request you to open a new thread .

Thanks
Eldert Grootenboer 160 Reputation points Microsoft Employee

2023-10-27T17:05:23.4966667+00:00

@Carlton Patterson I am the Program Manager for Service Bus, and have an additional question to the answers already given below. As the product group, we would like to know more about the scenarios where you would like to use this functionality. If you would like to share this information, feel free to either respond here or send us an email on servicebuspm@microsoft.com.

1 answer

MartinJaffer-MSFT 26,011 Reputation points

2022-02-07T23:18:30.663+00:00
Hello and welcome to Microsoft Q&A @Carlton Patterson .

It is my understanding you want to use Databricks to receive messages from Azure Service Bus queue and write them to some form of storage (Azure Blob storage / Azure Data Lake Gen2). YOu have your connection string, but are uncertain of next steps.

If you are using a Python / PySpark notebook, then you might want to look into the Python example script for receiving a message from a queue, and iterating over a queue. I found the examples here. There are also Java versions.

There are directions for connecting to Data Lake Gen 2 here.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Please sign in to rate this answer.
Carlton Patterson 741 Reputation points

2022-02-12T15:04:26.793+00:00

Hi MartinJaffer,

Thanks for reaching out.

I have used the link you suggested. However, the solution doesnt' show you have to write the data to Azure Blob or ADLS.

Would you agree?

MartinJaffer-MSFT 26,011 Reputation points

2022-02-23T17:43:49.367+00:00

@ CarltonPatterson-0401

For Azure Blob, you can mount the drive with this code:

dbutils.fs.mount( source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net", mount_point = "/mnt/<mount-name>", extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

For Data Lake Gen 1 you can mount or use service principal

val configs = Map( "<prefix>.oauth2.access.token.provider.type" -> "ClientCredential", "<prefix>.oauth2.client.id" -> "<application-id>", "<prefix>.oauth2.credential" -> dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-service-credential>"), "<prefix>.oauth2.refresh.url" -> "https://login.microsoftonline.com/<directory-id>/oauth2/token") // Optionally, you can add <directory-name> to the source URI of your mount point. dbutils.fs.mount( source = "adl://<storage-resource>.azuredatalakestore.net/<directory-name>", mountPoint = "/mnt/<mount-name>", extraConfigs = configs)
Sign in to comment