Performance issue in copy data activity with low throughput

Raj0125 511 Reputation points
2022-11-23T09:27:36.047+00:00

Hi All,

The problem is that the copy activity is very slow. It take about more than 3 hours to read 15GB and write 15 GB data.
The throughput is maximum 1.967 MB/s.
My configuration:
DIU units (in copy activity setting): 4
Used Parallel copies :1
source and sink are in same region
Service Tier: Standard(For workloads and typical performance requirements).

Can Anyone please help me with this how to handle this.

Please note past 2 days onwards facing this slowness with less through put before its completed successfully with in 50 mins with the through put maximum of 4.527 MB/s

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,145 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Samy Abdul 3,366 Reputation points
    2022-11-23T10:48:41.073+00:00

    Hi @Raj0125 , you could please looked in to these common measures to resolve

    the performance, such as :

    Increasing DIU
    Increasing degree parallelism
    Increasing concurrency of pipeline
    Splitting single file (smaller files)
    Set max rows per file

    Also, below threads should be quite useful. Thanks

    https://learn.microsoft.com/en-us/answers/questions/180644/how-to-optimize-azure-data-factorypipeline.htm

    https://learn.microsoft.com/en-us/answers/questions/36323/adf-performance-troubleshooting.htm

    https://github.com/Huachao/azure-content/blob/master/articles/data-factory/data-factory-copy-activity-performance.md

    https://stackoverflow.com/questions/70531362/how-to-increase-performance-of-azure-data-factory-pipeline