Storing message queues and process through ADF

Question

Hi All,

I need some help in finding resources that would help me build the following.

#1. Az Function - This will be used as endpoint for webhook to capture payloads from 3rd party vendor and append to the Queue Storage.

#2. ADF Pipeline - To consume the messages from the Queue's to the Az SQL Database.

Any best practices while designing the above requirement like storage design & logging.

Thank you!

Answer 1

Hi @PS ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question.

As I understand your issue , you want to accept 3rd party payload and store the output in queue storage and leverage ADF to further ingest it into SQL database. Please let me know if that is not the ask here.

• You can write Az function code and create new Azure Queue storage messages by setting up an output binding . Refer to this documentation: Azure Queue storage output bindings for Azure Functions .
• You can consider using web activity in ADF pipeline and use Az Queue storage API to retrieve the data and use copy activity to write the data into sql database. Kindly refer: Get message from Azure Queue using Web Activity in Azure Data Factory For this task, you can also leverage logic app which has queue storage connector that can be used to fetch the data and then use sql connector to copy the data . Please refer to : Azure Queue Storage

---

Hope it helps. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'