Copy Azure Service Bus messages in Queue to Data Lake via Databricks

Carlton Patterson 741 Reputation points
2022-02-04T18:56:58.727+00:00

I am trying to connect Azure Service Bus to Databricks and send messages in Azure Service Bus to data lake.

I have successfully connected Databricks to Service Bus via ConnectionString, but I'm not sure how to move messages in Service Queue to Lake via Databricks.

Can someone provide some guidance

Azure Service Bus
Azure Service Bus
An Azure service that provides cloud messaging as a service and hybrid integration.
558 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,036 Reputation points
    2022-02-07T23:18:30.663+00:00

    Hello and welcome to Microsoft Q&A @Carlton Patterson .

    It is my understanding you want to use Databricks to receive messages from Azure Service Bus queue and write them to some form of storage (Azure Blob storage / Azure Data Lake Gen2). YOu have your connection string, but are uncertain of next steps.

    If you are using a Python / PySpark notebook, then you might want to look into the Python example script for receiving a message from a queue, and iterating over a queue. I found the examples here. There are also Java versions.

    There are directions for connecting to Data Lake Gen 2 here.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how