Is there a way to connect Azure Batch to Storage Account over a private network when running via ADF?

Question

I have a Storage Account which is set to be available only in the private network.

Similarly, I have a Batch Account and a Data Factory that are set to private connectivity as well.

Now when I try to run a batch task from the data factory, I got to understand that Azure Managed Integration Runtime does not support private connectivity with Batch. Hence, I whitelisted the publicly available public IPs for Data Factory in my region (Central India) at the Batch Account.

Now when I try to run a batch task from data factory, I faced a lot of issues accessing the storage from batch. On asking around, I got to understand that ADF is the intermediary between the Storage Account and the Batch Account. Therefore, since ADF interacts with the Batch Account through the public network, it would interact with the storage account over the public network as well. Hence, the data from storage account to batch will be transferred over the public network.

Transferring data over the public network will be a compliance issue as we want the data to be transferred over the private network only.

Is there a way to force Batch Account to connect with the storage account over the private network when running tasks from the data factory? We do not wish to use the Self-hosted Integration Runtime because of the large amount of management involved with manual scaling and additional costs involved.

Answer 1

Hi Prashant,

Based on my research, it might not be possible to achieve this without using Self-hosted IR in ADF at the moment.
Would your customer be open to using Azure functions for triggering batch jobs? If yes, I can share some high-level steps and document links to help by requesting some additional information for clarification.

Thanks, Karni.G If you find my answer helpful, please consider marking it as the ‘Answer’ and giving it an ‘Upvote’ using the thumbs-up option. This can also benefit other community members who may have the same question.

Answer 2

In your case, even though the connection between MIR and Batch Service might be via public internet, you can still set up a secure private connection between the Azure Batch service and Azure Storage service using Private Endpoints. This would ensure that the data traffic from Azure Batch to Azure Storage remains on Microsoft's backbone network, which enhances security. Azure Batch does support private endpoint connections. It can connect to a Storage Account via a private endpoint, which is a network interface within your virtual network that connects you privately and securely to a service powered by Azure Private Link. This setup ensures that traffic between the Batch account and the Storage account traverses over your virtual network and a private link on the Microsoft backbone network, eliminating exposure from the public internet. Here are some steps to set this up: Create a private endpoint for your storage account in the relevant virtual network. Configure the private endpoint connection in your storage account networking settings. Use the private endpoint (FQDN) to connect from your Batch account.

Please remember, while setting up private endpoints, you must ensure that your networking configurations allow traffic between your Batch account and the Storage account over the chosen subnet. This should ensure your data is transferred privately within your network. If you're still facing issues, I'd recommend contacting Azure Support for more specific troubleshooting assistance.