How to set an Azure Data Factory WebHook activity to use a specified self-hosted integration runtime?

Question

How to set an Azure Data Factory WebHook activity to use a specified self-hosted integration runtime?

Platt, Jason 20

I have a Azure Data Factory pipeline that copies data from a local database to the disk of a self-hosted integration runtime. Then I need to run a local process on it, and I'd like to use the WebHook Activity, however I keep getting the error "Activity failed because error occurred while processing the request: Localhost address only allowed on Self-Hosted Integration Runtimes". The WebHook web dialog doesn't have a place to specify which integration runtime it is to run on, and manually adding the "connectVia" json block gets deleted by the parser. Help?

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-26T05:05:28.7733333+00:00
@Platt, Jason The Asynchronous Request-Reply pattern is used to decouple backend processing from a frontend host, where backend processing needs to be asynchronous, but the frontend still needs a clear response. This pattern is particularly useful when the work done by the backend may be long-running.

One solution to implement this pattern is to use HTTP polling. Here’s how it works:

The client application makes a synchronous call to the API, triggering a long-running operation on the backend.

The API responds as quickly as possible, acknowledging the request has been received. This is usually in the form of a HTTP 202 (Accepted) status code.

The response contains an endpoint that the client can poll to check the status of the long-running operation.

The API processes the background work, which could be in the form of sending a message to a queue.

When a successful call is made to the status endpoint, it should return HTTP 200 (Ok) as well as an update on the status of the background process work.

Once the work is complete, the status endpoint can either return a resource that shows the work is complete or redirect to another resource URL.

For more detailed information, you can refer to the Azure Architecture Center’s guide on the Asynchronous Request-Reply pattern and this DEV Community article that provides a practical explanation of the pattern. Remember, the implementation of this pattern can vary based on your specific use case and requirements. Always consider factors such as the nature of your backend processing, the expected response times, and the infrastructure you have in place.
Platt, Jason 20 Reputation points

2024-03-07T23:35:22.3633333+00:00

I've solved my issue by implementing the request response pattern, which seems to be working quite well. It also works in the on-prem environment.

Accepted answer

0 additional answers

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-26T05:05:28.7733333+00:00

@Platt, Jason The Asynchronous Request-Reply pattern is used to decouple backend processing from a frontend host, where backend processing needs to be asynchronous, but the frontend still needs a clear response. This pattern is particularly useful when the work done by the backend may be long-running.

One solution to implement this pattern is to use HTTP polling. Here’s how it works:

The client application makes a synchronous call to the API, triggering a long-running operation on the backend.

The API responds as quickly as possible, acknowledging the request has been received. This is usually in the form of a HTTP 202 (Accepted) status code.

The response contains an endpoint that the client can poll to check the status of the long-running operation.

The API processes the background work, which could be in the form of sending a message to a queue.

When a successful call is made to the status endpoint, it should return HTTP 200 (Ok) as well as an update on the status of the background process work.

Once the work is complete, the status endpoint can either return a resource that shows the work is complete or redirect to another resource URL.

For more detailed information, you can refer to the Azure Architecture Center’s guide on the Asynchronous Request-Reply pattern and this DEV Community article that provides a practical explanation of the pattern. Remember, the implementation of this pattern can vary based on your specific use case and requirements. Always consider factors such as the nature of your backend processing, the expected response times, and the infrastructure you have in place.
Platt, Jason 20 Reputation points

2024-03-07T23:35:22.3633333+00:00

I've solved my issue by implementing the request response pattern, which seems to be working quite well. It also works in the on-prem environment.

Answer 1

The Web Activity and Webhook Activity are designed to send HTTP requests to a public endpoint on the internet, not to a local network only accessible via your self-hosted integration runtime. Unfortunately, as far as I understand, there's no way to configure Web Activity or Webhook Activity to run on a specific integration runtime. They will always use Azure's infrastructure to send the HTTP request, so they can't communicate with a localhost address in your private network. Here are alternative solutions: Expose your localhost service: Expose your local process as a service accessible through the internet using tools like ngrok or localtunnel. But this method might have security implications. Use Azure Function: Create an Azure Function that triggers your local process, and call this function from your Data Factory pipeline. You'd need to secure your Azure Function properly if it's going to start processes on your local machine. Use Custom .NET activity: If the process you need to run is a script or an executable file, consider creating a custom .NET activity in Azure Batch. This would require more setup and potentially cost but would run within your control. Use Logic Apps: Logic Apps have more connectors than Azure Data Factory and may be able to solve your problem.

Remember to evaluate each approach considering your overall architecture, security, and compliance requirements. Consult your system administrator or IT support for specific advice. My advice is based on my theoretical understanding of how the components work, I have not had a chance to implement it myself.

Platt, Jason 20 Reputation points

2024-02-21T16:29:20.8166667+00:00

The Web Activity lets you choose the integration runtime to use, and works fine calling private network resources from a self-hosted integration runtime. The WebHook seems to be missing that functionality, yet produces a message indicating that it should. Looks like I will have to resort to performing polling for the status :(
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-23T12:46:58.6133333+00:00

@Platt, Jason We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Platt, Jason 20 Reputation points

2024-02-23T16:10:40.6866667+00:00

@phemanth I'm attempting to use the async-request-reply pattern, however the documentation that I have found is a little lacking.

Share via

How to set an Azure Data Factory WebHook activity to use a specified self-hosted integration runtime?

0 additional answers

Your answer