Mapping Data Flow Azure data factory

Question

I am flattening a Json File to CSV file using mapping data flow in Azure data factory, the issue is that file is placed in storage container which is behind firewall. As it looks Only Auto Resolve integration runtime is supported in Mapping Data flow, and I cant create a linked service with AutoResolve integration runtime to connect to firewalled storage container, Am I missing something here, do we have a workaround for this.

Answer 1

Hi @Anonymous ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

As per my understanding you have a storage account with firewall restrictions and you would like to connect to it from mapping data flow using Azure IR. Please correct me if I'm wrong anywhere.

If that's the case then in order to establish a secure connection from Mapping data flow to your Storage account with firewall restrictions, you will have to create an Azure integration runtime within a Data Factory managed virtual network, the integration runtime is provisioned with the managed virtual network. It uses private endpoints to securely connect to supported data stores.

Managed private endpoints are private endpoints created in the Data Factory managed virtual network that establishes a private link to Azure resources. Data Factory supports private links. You can use Azure private link to access Azure platform as a service (PaaS) services like Azure Storage. A private endpoint uses a private IP address in the managed virtual network to effectively bring the service into it. Private endpoints are mapped to a specific resource in Azure and not the entire service.

Here is a public document which has clear steps on how to build a mapping data flow pipeline using managed vNet and private endpoints to transform data securely: Transform data securely by using mapping data flow

I would highly recommend going through below documentation before you try out this feature.

MS Doc: Azure Data Factory managed virtual network

Hope this helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.