Azure Data Factory and Azure Kubernetes Services - Establish a secure connection between the two services in different subscriptions

Question

Scenario: Use a function activity or a web service activity in a data factory pipeline to communicate with a private web service deployed on Azure Kubernetes Services in a different subscription within our organization

Reference URLs :

1.https://learn.microsoft.com/en-us/answers/questions/326091/azure-date-factory-managed-virtual-network-can-you.html
2.https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint
3.https://learn.microsoft.com/en-us/answers/questions/625573/2-way-traffic-with-azure-private-link.html

From the first two references posted above, It can be understood that ADF does not offer a straightforward way to peer with other virtual networks. Using an Azure-managed Vnet and a private endpoint in the Data factory, you can establish a private link to a Paas service such as blob storage or Azure SQL.

From the third reference, a private end point connection is only one way connection to a Paas service.

I would like to send a request to a web service deployed in a different subscription and receive a response back. This is a two-way connection. Is taking the private endpoint route the best approach to implement this use case? Please suggest how I can implement this scenario.

Answer 1

Hello @Govindu, Bipin (RIS-SAN) and welcome to Microsoft Q&A.

Azure Data Factory is the initiator in almost all activities. That is, when Copy activity happens, Data Factory reaches out to the source and sink, when Web activity happens, Data Factory makes the call. Data Factory does not wait for the data source or web service to call Data Factory. When Data Factory makes the request or call, a response is expected like in a telephone call. When on the telephone, you do not need to dial the other person back while the call is ongoing. In this manner, the connection does not need to be two-way.

You could also try a Self-Hosted Integration Runtime, and push the Web activity through that.