How do I import or load data from external webhook(external system) into Azure

Question

How do I import or load data from external webhook(external system) into Azure

Kesavan R 1

Hi Experts,

We have a live chat platform which has the ability to push the data to any Endpoint URL (in json format) whenever any events(like chat conversation started, completed etc.,) gets triggered in their system, with the help of webhooks that is created at their side. The idea is to create those Endpoint URL in Azure so that we can store the data in storage containers whenever the data is posted/triggered by webhooks from their system.

As shown in below snip, this webhook(created at their side) will push the data to the mentioned Endpoint URL(required field) whenever the selected events gets triggered. I am new to Azure and could not figure out how can we consume/load the data from this webhook into Azure storage containers.

User's image

I just tried to give the URL of the existing azure storage container in this "URL" required field while creating webhooks at their side, but it didn't work, got the below message when the event got triggered at their side. User's image

Could you pls let me know how can we load the data from external webhooks into Azure ?

MartinJaffer-MSFT 26,236 Reputation points

2023-02-06T19:31:08.9066667+00:00

@Kesavan R Hello and welcome to Microsoft Q&A

As I understand you have a third party tool which makes calls to webhooks and you want to push the data to Azure Storage.

For this I recommend Azure Logic App or Azure Functions, maybe API Management?

While Data Factory has a "webhook activity", it is not the right tool for this job. Data Factory expects to be the initiator.

If you are comfortable writing code, use Azure Functions.

If you do not know how to code, and prefer a visual interface, use Logic Apps. Logic App also has connector for Azure Storage, so that should be easiest to set up.

If your application has a very high throughput (lots of activity) you may want to leverage something like Event Hubs, Service Bus, or Load balancer.
Kesavan R 1 Reputation point

2023-02-13T12:28:35.49+00:00

Hi @MartinJaffer-MSFT ,

Thanks a lot for the answer. Logic App worked in the sandbox environment. We would need to try it in the PROD environment to see how it behaves. We are concerned about the throughput bcos there will be too many live chats in the PROD that will trigger the Logic App multiple times per minute.

Could you pls let us know regarding the cost of Logic App so that we can make an assessment with the volume of chats that might be expected in the PROD env.?
Kesavan R 1 Reputation point

2023-02-13T12:50:23.42+00:00

Hi @MartinJaffer-MSFT ,

Logic App worked well in the sandbox environment. We would need to try it in the PROD env to see how it behaves. We are concerned about the cost bcos there would be too many live chats happening in PROD and that would trigger the Logic App multiple times within a minute and not sure how would that affect. Could you pls let me know the cost factor for Logic App ? Will the other options that you mentioned - (Event Hubs, Service Bus, or Load balancer) be more suitable for this use case ? We have not explored those options yet
MartinJaffer-MSFT 26,236 Reputation points

2023-02-17T20:48:34.16+00:00

@Kesavan R I'm not expert in Logic apps or Load balancer.

There are multiple pricing plans for Logic App.

Event Hubs / Service bus is really good when used as a buffer. It is basically a giant queue, supporting multiple readers and multiple writers. The readers would be the Logic App or whatever you choose. The writer would be your chats streaming messages in.

Load Balancer is on the network level, spreading out load. It would take all the incoming traffic and redirect to multiple instances of your Logic App or whatever you choose.

For more on architecture patters on Logic Apps, check here.

1 answer

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2023-02-06T19:31:08.9066667+00:00

@Kesavan R Hello and welcome to Microsoft Q&A

As I understand you have a third party tool which makes calls to webhooks and you want to push the data to Azure Storage.

For this I recommend Azure Logic App or Azure Functions, maybe API Management?

While Data Factory has a "webhook activity", it is not the right tool for this job. Data Factory expects to be the initiator.

If you are comfortable writing code, use Azure Functions.

If you do not know how to code, and prefer a visual interface, use Logic Apps. Logic App also has connector for Azure Storage, so that should be easiest to set up.

If your application has a very high throughput (lots of activity) you may want to leverage something like Event Hubs, Service Bus, or Load balancer.
Kesavan R 1 Reputation point

2023-02-13T12:28:35.49+00:00

Hi @MartinJaffer-MSFT ,

Thanks a lot for the answer. Logic App worked in the sandbox environment. We would need to try it in the PROD environment to see how it behaves. We are concerned about the throughput bcos there will be too many live chats in the PROD that will trigger the Logic App multiple times per minute.

Could you pls let us know regarding the cost of Logic App so that we can make an assessment with the volume of chats that might be expected in the PROD env.?
Kesavan R 1 Reputation point

2023-02-13T12:50:23.42+00:00

Hi @MartinJaffer-MSFT ,

Logic App worked well in the sandbox environment. We would need to try it in the PROD env to see how it behaves. We are concerned about the cost bcos there would be too many live chats happening in PROD and that would trigger the Logic App multiple times within a minute and not sure how would that affect. Could you pls let me know the cost factor for Logic App ? Will the other options that you mentioned - (Event Hubs, Service Bus, or Load balancer) be more suitable for this use case ? We have not explored those options yet
MartinJaffer-MSFT 26,236 Reputation points

2023-02-17T20:48:34.16+00:00

@Kesavan R I'm not expert in Logic apps or Load balancer.

There are multiple pricing plans for Logic App.

Event Hubs / Service bus is really good when used as a buffer. It is basically a giant queue, supporting multiple readers and multiple writers. The readers would be the Logic App or whatever you choose. The writer would be your chats streaming messages in.

Load Balancer is on the network level, spreading out load. It would take all the incoming traffic and redirect to multiple instances of your Logic App or whatever you choose.

For more on architecture patters on Logic Apps, check here.

Answer 1

To consume the data from the webhook and store it in an Azure storage container, you can follow these steps:

Create an Azure Functions App: You can create an Azure Functions app to act as the Endpoint URL. This Azure Function will receive the data from the webhook and process it.
Write the code for the Azure Function: You need to write the code for the Azure Function to receive the data from the webhook and store it in an Azure storage container. You can use the Azure Functions bindings to write code that can receive and process the incoming data.
Create an Azure storage account: You need to create an Azure storage account to store the data that is posted to the Azure Function by the webhook. You can create a Blob storage account or a Table storage account depending on your requirements.
Configure the Azure Function to store data in the storage account: You need to configure the Azure Function to store the data that is posted to it by the webhook in the Azure storage account. You can use the Azure Functions bindings to write code that can store the data in the storage account.
Test the setup: After you have completed all the steps, you can test the setup by posting data from the webhook to the Azure Function. You can verify that the data is being stored in the Azure storage account.

This is a high-level overview of the steps you need to follow to consume the data from the webhook and store it in an Azure storage container. You can find more information and detailed instructions on how to perform each step in the Azure documentation.

Share via

How do I import or load data from external webhook(external system) into Azure

1 answer

Your answer