Ensuring Private Connectivity for Cross-Cloud Data Transfers Using STS and Azure Data Factory

Question

Hello,

I am working on a project where Google Cloud and Azure are connected via a private connection (Cloud Interconnect/ExpressRoute). I need to ensure that data transfers between Google Cloud Storage and Azure Storage happen over this private connection, rather than using the public internet.

Specifically, I have two scenarios:

Pulling data from Azure Storage to Google Cloud Storage using Google Cloud's Storage Transfer Service (STS): How can I ensure that STS uses the private connection between Google Cloud and Azure for the transfer, as STS typically uses public APIs and does not inherently support private connectivity out of the box?

Pulling data from Google Cloud Storage to Azure Storage using Azure Data Factory: How can I ensure that Azure Data Factory leverages the private connection (via ExpressRoute or Cloud Interconnect) to transfer data from Google Cloud to Azure, rather than using the public internet?

What configurations or steps should be implemented to ensure both transfers occur securely over the private connection?

I appreciate any insights or guidance on how to set up private connectivity for both of these data transfer scenarios.

Answer 1

Hi @Nikunj Patel

Greetings! Welcome to Microsoft Q&A Forum. Thanks for posting your query here!

Firstly, apologies for delay in response.

I understand that you would like to have private connection for data transfer between Google Cloud Storage and Azure Storage. Please make sure that private connectivity for cross cloud data transfers using STS and Azure Data Factory which involves few steps and configuration setup to each service.

Pulling data from Azure Storage to Google Cloud Storage using Google Cloud's Storage Transfer Service (STS):

• Please make sure to setup a private connection that cloud interconnect is properly set up between Google Cloud and Azure Storage. To ensure that STS utilizes the private connection, please set up private API endpoints in Cloud Interconnect. This involves configuring the private endpoint in Google Cloud and verifying that the necessary permissions are in place.
• Please ensure that the Storage Transfer Service is set up to use the private endpoint for data transfers. This may require configuring specific network rules and verifying that the transfer agent is properly set to utilize the private connection.

Pulling data from Google Cloud Storage to Azure Storage using Azure Data Factory:

• Creating a Private Endpoint in Azure Data Factory: Establish a private endpoint in Azure Data Factory to ensure data transfers utilize a private connection. This process involves setting up a private endpoint within your virtual network and configuring it to interact with Google Cloud Storage.
• Configure Azure Data Factory: Please set up the required pipelines and linked services in Azure Data Factory to utilize the private endpoint. Ensure that the self-hosted integration runtime (IR) is configured to use the private network for data transfers.
• Turn on Managed Virtual Network: Make sure to use the managed virtual network feature in Azure Data Factory for secure communication between your networks. This will keep your data transfers safe from public network attacks.

By following these steps to make sure both data transfers are secure over your private connection instead of the public internet.

I hope this helps, please feel free to reach us for any further assistance. We will be glad to assist you closely.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. which might be beneficial to other community members reading this thread.