How to call Rest API of a SaaS product from an ADF connected to a private virtual network?

Question

How to call Rest API of a SaaS product from an ADF connected to a private virtual network?

Pratim Das, Partha C 346

Hi Champs?

I have a basic question: How to call Rest API of a SaaS product from an ADF connected to a private virtual network? Which type of Integration Runtime should I use?

Regards,

Partha

Pratim Das, Partha C 346 Reputation points

2023-11-27T02:01:54.9233333+00:00

Hi,

Any help in this regard is highly appreciated

Regards,

Partha
Harishga 6,000 Reputation points Microsoft External Staff

2023-11-27T06:39:10.69+00:00

Hi@Pratim Das, Partha C ,
Welcome to Microsoft Q&A platform and thanks for posting your question here.

When selecting the appropriate type of integration runtime, there are various factors to consider, such as the type of SaaS product being connected to, whether it is publicly accessible or behind a firewall or in a private network, the level of security required during data transmission, and the level of maintenance that can be provided. It is also important to consider the locations of the integration runtime and data store.

In order to call a REST API of a SaaS product from an ADF that is connected to a private virtual network, a self-hosted integration runtime is necessary. This is due to the possibility of the REST API endpoint being inaccessible from the public internet, and a self-hosted integration runtime must be installed and configured on a machine or virtual machine that can access the endpoint. Another option is to use a web activity in ADF to make the REST API call, specifying the authentication method and request method (GET, POST, PUT, or PATCH) as required.

Reference link:

https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory

I hope this information helps you. Let me know if you have any further questions or concerns.
Pratim Das, Partha C 346 Reputation points

2023-11-27T07:04:08.7833333+00:00

Thanks @Harishga for your response. In my case my ADF is within a Vent with no public access. In that case How I can access a REST api which is available over internet. Could you please elaborate?
Harishga 6,000 Reputation points Microsoft External Staff

2023-11-27T08:34:56.7166667+00:00
Hi@Pratim Das, Partha C
To access a REST API that is available over the internet from an Azure Data Factory within a virtual network with no public access, you need to use Azure Private Link and create a private endpoint for the REST API service. Azure Private Link allows you to securely connect to Azure SaaS services from your virtual network using a private IP address. This way, you can avoid exposing your data to the public internet and protect it from data exfiltration risks.

To create a private endpoint for the REST API service, you need to follow these steps:

Enable managed virtual network in your data factory. You can do this either during the creation of the data factory or in the integration runtime settings.

Create a REST linked service in your data factory using the base URL of the REST API service and the authentication type that the service requires.

Create a REST dataset in your data factory using the relative URL of the REST API resource that you want to access.

Create a managed private endpoint in your data factory for the REST linked service.

Approve the private endpoint connection request from the REST API service provider.

Test the connectivity between your data factory and the REST API service using a web activity or a copy activity in a pipeline.

Reference
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-private-link
https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint

I hope this information helps you. Let me know if you have any further questions or concerns.
Pratim Das, Partha C 346 Reputation points

2023-11-28T04:30:30.22+00:00
Thanks @Harishga for your detailed explanation. I need two more suggestions

Which IR do you recommend? SHIR or Azure IR

How can I create private end point for individual Linked service assuming I already have PE for the ADF

Note: My REST API is exposed from a SaaS outside azure and accesible over public internet. Is your solution still valid for that scenario?

Regards,

Partha
Harishga 6,000 Reputation points Microsoft External Staff

2023-11-28T12:45:08.93+00:00
Hi@Pratim Das, Partha C
My recommendation for the integration runtime would be to use a self-hosted integration runtime (SHIR) since your ADF is within a VNet with no public access. This will allow you to install and configure the integration runtime on a machine or virtual machine that can access the REST API endpoint.

To create a private endpoint for an individual linked service, you can follow these steps:

Navigate to the Azure portal and open your data factory.

Click on the "Author & Monitor" button to open the ADF authoring UI.

Click on the "Manage" tab and select "Linked services".

Select the linked service for which you want to create a private endpoint.

Click on the "Private endpoint connections" tab and then click on the "New" button.

Select the virtual network and subnet where you want to create the private endpoint.

Click on the "Review + create" button and then click on the "Create" button to create the private endpoint.

Regarding your scenario where the REST API is exposed from a SaaS outside Azure and accessible over the public internet, the solution I provided earlier is still valid. You can create a private endpoint for the REST API service using Azure Private Link to securely connect to the service from your virtual network using a private IP address.

Your answer

Pratim Das, Partha C 346 Reputation points

2023-11-27T02:01:54.9233333+00:00

Hi,

Any help in this regard is highly appreciated

Regards,

Partha
Harishga 6,000 Reputation points Microsoft External Staff

2023-11-27T06:39:10.69+00:00

Hi@Pratim Das, Partha C ,
Welcome to Microsoft Q&A platform and thanks for posting your question here.

When selecting the appropriate type of integration runtime, there are various factors to consider, such as the type of SaaS product being connected to, whether it is publicly accessible or behind a firewall or in a private network, the level of security required during data transmission, and the level of maintenance that can be provided. It is also important to consider the locations of the integration runtime and data store.

In order to call a REST API of a SaaS product from an ADF that is connected to a private virtual network, a self-hosted integration runtime is necessary. This is due to the possibility of the REST API endpoint being inaccessible from the public internet, and a self-hosted integration runtime must be installed and configured on a machine or virtual machine that can access the endpoint. Another option is to use a web activity in ADF to make the REST API call, specifying the authentication method and request method (GET, POST, PUT, or PATCH) as required.

Reference link:

https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory

I hope this information helps you. Let me know if you have any further questions or concerns.
Pratim Das, Partha C 346 Reputation points

2023-11-27T07:04:08.7833333+00:00

Thanks @Harishga for your response. In my case my ADF is within a Vent with no public access. In that case How I can access a REST api which is available over internet. Could you please elaborate?
Harishga 6,000 Reputation points Microsoft External Staff

2023-11-27T08:34:56.7166667+00:00

Hi@Pratim Das, Partha C
To access a REST API that is available over the internet from an Azure Data Factory within a virtual network with no public access, you need to use Azure Private Link and create a private endpoint for the REST API service. Azure Private Link allows you to securely connect to Azure SaaS services from your virtual network using a private IP address. This way, you can avoid exposing your data to the public internet and protect it from data exfiltration risks.

To create a private endpoint for the REST API service, you need to follow these steps:

Enable managed virtual network in your data factory. You can do this either during the creation of the data factory or in the integration runtime settings.

Create a REST linked service in your data factory using the base URL of the REST API service and the authentication type that the service requires.

Create a REST dataset in your data factory using the relative URL of the REST API resource that you want to access.

Create a managed private endpoint in your data factory for the REST linked service.

Approve the private endpoint connection request from the REST API service provider.

Test the connectivity between your data factory and the REST API service using a web activity or a copy activity in a pipeline.

Reference
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-private-link
https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint

I hope this information helps you. Let me know if you have any further questions or concerns.
Pratim Das, Partha C 346 Reputation points

2023-11-28T04:30:30.22+00:00

Thanks @Harishga for your detailed explanation. I need two more suggestions

Which IR do you recommend? SHIR or Azure IR

How can I create private end point for individual Linked service assuming I already have PE for the ADF

Note: My REST API is exposed from a SaaS outside azure and accesible over public internet. Is your solution still valid for that scenario?

Regards,

Partha
Harishga 6,000 Reputation points Microsoft External Staff

2023-11-28T12:45:08.93+00:00

Hi@Pratim Das, Partha C
My recommendation for the integration runtime would be to use a self-hosted integration runtime (SHIR) since your ADF is within a VNet with no public access. This will allow you to install and configure the integration runtime on a machine or virtual machine that can access the REST API endpoint.

To create a private endpoint for an individual linked service, you can follow these steps:

Navigate to the Azure portal and open your data factory.

Click on the "Author & Monitor" button to open the ADF authoring UI.

Click on the "Manage" tab and select "Linked services".

Select the linked service for which you want to create a private endpoint.

Click on the "Private endpoint connections" tab and then click on the "New" button.

Select the virtual network and subnet where you want to create the private endpoint.

Click on the "Review + create" button and then click on the "Create" button to create the private endpoint.

Regarding your scenario where the REST API is exposed from a SaaS outside Azure and accessible over the public internet, the solution I provided earlier is still valid. You can create a private endpoint for the REST API service using Azure Private Link to securely connect to the service from your virtual network using a private IP address.

Share via

How to call Rest API of a SaaS product from an ADF connected to a private virtual network?

Your answer