Send large amount of data to external systems

Question

Hi friends, we have a requirement to send a large amount of data in gigabytes from Azure to third-party external systems. The data needs to be sent to both azure and API public endpoint methods. How to send this securely without latency? What are the azure services need to be used?

Answer 1

In my opinion, you can have 3 alternatives :

• ADF
• Azure Synapse
• Azure Blob

For the first two options, you can use one of them separately or to complete each other as you didn't provide more information.

Your solution using ADF :

You can start copying data between various data stores in a secure and reliable manner. For large datasets, ADF provides features like parallel copying and staged copying (using Azure Blob Storage as a temporary location) to enhance throughput and manage large volumes of data effectively.

Also the Integration Runtime offers data integration capabilities across different network environments. For instance, a Self-hosted Integration Runtime can be used to transfer data securely from Azure to on-premises systems or to other clouds.

Your solution using Azure Synapse :

Synapse can integrate with Azure Data Factory, allowing the orchestration of data workflows that include large-scale data processing tasks.

For security matters, you have the managed private endpoints, which can securely connect to other Azure services or external services via Azure Private Link.

Your solution using Azure Synapse :

You can use it as a staging area for large files. From there, you can use Azure functions or other mechanisms to trigger data movement to external APIs.

Tools like AzCopy can efficiently transfer data to and from Azure Blob Storage. It supports high-performance data transfers by optimizing for throughput.

What to keep in mind to minimize latency and maximize transfer speeds ?

You can use Azure ExpressRoute which establishes a direct, private connection between Azure and your on-premises infrastructure or other clouds. This bypasses the public internet, reducing latency and increasing security.

For geographically dispersed systems, you can use Azure CDN to help reduce load times and save bandwidth, improving the user experience for distributed clients.