Data Transfer solution

Question

Hello,

We would like to transfer files from below source and destination using a secure connection. My understanding is that we can use Express Route and ADF SHIR. The old process uses SFTP jobs to transfer the files.

Could you please share the best possible approach to transfer files based on the below conditions.

What are the steps to configure the connectivity between source and destination and tranfer of files ?

In case of linux vm where SHIR is possibly not configurable how can we transfer the files ?

1.On-Prem > > Azure Storage

2.Azure Storage >> On-prem

3.External Vendor >> Azure Storage

Answer 1

@Niren Adhikary - Thanks for the question and using MS Q&A platform.

To transfer files securely between the source and destination, you can use Azure Data Factory (ADF) and Azure ExpressRoute. Here are the steps to configure the connectivity and transfer files:

• Configure ExpressRoute: You need to configure ExpressRoute to establish a private, dedicated connection between your on-premises network and Azure. This will ensure that the data transfer is secure and reliable.
• Configure Azure Storage: You need to create an Azure Storage account and configure it to allow access from your on-premises network. You can use Azure Private Link to securely access the storage account over the ExpressRoute connection.
• Configure ADF: You need to create an ADF pipeline to transfer the files between the source and destination. You can use the Azure Blob Storage connector to read and write files to Azure Storage. You can also use the Self-Hosted Integration Runtime (SHIR) to securely transfer files between the on-premises network and Azure Storage.
• Configure SFTP: If you need to transfer files using SFTP, you can use the SFTP connector in ADF to securely transfer files between the source and destination. In case of a Linux VM where SHIR is not configurable, you can use the Azure File Sync service to synchronize files between the on-premises network and Azure Storage. This service allows you to create a sync group to synchronize files between a file server and an Azure file share. You can also use the Azure Storage Explorer tool to transfer files between the on-premises network and Azure Storage.

For transferring files from an external vendor to Azure Storage, you can use the Azure Data Share service. This service allows you to securely share data with external partners and customers. You can create a data share to share files with the external vendor and configure the access rights to ensure that the data is secure. The external vendor can then use the Azure Blob Storage connector to read and write files to the shared data.

Additonal question: so we the following requirement and we want to use a secure method or pattern. 1. External source to Azure Blobstorage. Can we use ADF SHIR? Can we use Express route for External Vendor. Alternatively, how can we push the files from external source to Azure using SFTP jobs. 2. Azure Data Lake to on-prem Linux path. Can I have a detail steps on the above.

Yes, you can use Azure Data Factory (ADF) with Self-Hosted Integration Runtime (SHIR) to transfer files securely between an external source and Azure Blob Storage. SHIR allows you to securely transfer data between on-premises data stores and cloud data stores. You can install SHIR on a machine in your on-premises network and configure it to securely transfer data to and from Azure Blob Storage.

• If you want to use ExpressRoute for the external vendor, you can configure ExpressRoute to establish a private, dedicated connection between the external vendor's network and Azure. This will ensure that the data transfer is secure and reliable.
• Alternatively, you can use SFTP jobs to transfer files from the external source to Azure Blob Storage. You can use the SFTP connector in ADF to securely transfer files between the external source and Azure Blob Storage.
• To transfer files from Azure Data Lake to an on-premises Linux path, you can use ADF with SHIR. You can install SHIR on a machine in your on-premises network and configure it to securely transfer data from Azure Data Lake to the on-premises Linux path. You can use the Azure Data Lake Storage connector in ADF to read and write files to Azure Data Lake. For more details, refer to Choose the right integration runtime configuration for your scenario.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Hi Niren Adhikary

The Self Hosted Integration Runtime is currently available to install only on Windows systems

In case of Windows VM you can install SHIR over there

1. On-Prem > > Azure Storage

I am assuming the On-Prem location is a SFTP location, if yes then it can be easily pushed to Azure Storage using Azure Data Factory's Copy Activity

1. Azure Storage >> On-prem

This is reverse case of 1st case, still you can use Azure Data Factory's Copy Activity to configure Azure Storage as source (this will use Azure IR) and sink as SFTP (it will use SHIR)

1. External Vendor >> Azure Storage

It depends how the vendor is planning to push the data, it can be one of the following ways:

(a) If they are planning to push via their custom application, then you can create a service principal which should have write access on the particular container where the file is supposed to be pushed

(b) If they will be pushing files manually then a Shared Access Signature (SAS) can be created for the container where the data needs to be pushed

(c) They can use AzCopy as well

Whenever dealing with external vendors to push the data, make sure to restrict the access and provide minimum required access. Also, better to create a separate container and provide access to push the data there, then if needed, that data can be copied from that container to the main container using ADF copy activity