How to increase the throughput speed while data migration through ADF ?

Rahul 251 Reputation points
2023-07-12T18:04:17.57+00:00

Hi,

Question1- How to increase the network throughput speed in the below below architect , data migration from oracle (on-prem) to azure sql database?

Connection-1

Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->Sql server database.

Connection-2

Oracle (on-prem)-->ADFSHIR-->F5-->Express Route Gateway-->Azurefirewall-->Private End Point-->data Factory instance.

I am getting throughput speed- 2 -5 Mbps due to which the 1.5 TB data migration is taking 100 Hr , for sequential each table migration .

How i can improve the throughput speed?

I check above each components have got good good network bandwidth, but i am unable to identify which component is keeping the throughput speed low overall?

Please help what should be approach to improve throughput speed.

Oracle dynamic partioning, azure sql write batch size, parallel connection- All these logic i have already implemented in migration pipeline as well.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,209 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,536 Reputation points
    2023-07-13T03:42:29.9466667+00:00

    @Rahul - Thanks for the question and using MS Q&A platform.

    The following visual illustrates the guidelines to choose the various Azure data transfer tools depending upon the network bandwidth available for transfer, data size intended for transfer, and frequency of the transfer.

    Azure data transfer tools

    For more details, refer to Choose an Azure solution for data transfer.

    To increase the network throughput speed for data migration from Oracle (on-prem) to Azure SQL Database using Azure Data Factory, you can try the following approaches:

    Increase the number of parallel connections: You can increase the number of parallel connections in the Azure Data Factory pipeline to improve the throughput speed. This can be done by increasing the number of concurrent connections in the source and sink settings.

    • Increase the batch size: You can increase the batch size for writing data to Azure SQL Database to improve the throughput speed. This can be done by increasing the write batch size in the sink settings.
    • Use dynamic partitioning: You can use dynamic partitioning in the Azure Data Factory pipeline to partition the data into smaller chunks and process them in parallel. This can help improve the throughput speed by distributing the workload across multiple nodes.
    • Optimize the network configuration: You can optimize the network configuration by ensuring that all components in the data migration pipeline have sufficient bandwidth and are configured correctly. This can include optimizing the network settings for the Express Route Gateway, Azure Firewall, and Private Endpoint.
    • Use a dedicated network connection: You can use a dedicated network connection, such as a dedicated Express Route circuit, to improve the network throughput speed. This can help ensure that the network bandwidth is not shared with other applications or services.
    • Use a different migration tool: If the above approaches do not improve the throughput speed, you can consider using a different migration tool that is optimized for high-speed data migration, such as Azure Database Migration Service.

    It's important to note that the network throughput speed can be affected by a variety of factors, such as network latency, network congestion, and network configuration. You may need to experiment with different approaches to find the optimal configuration for your specific scenario.

    For more details, refer to Copy activity performance and scalability guide and Copy activity performance optimization features.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.