Copy Activity very slow for Azure data factory

Question

Copy Activity very slow for Azure data factory

Mayuri Shah 0

Hi All,

we are having issue where copy activity from Azure Table storage to data factory is taking over 50mins on daily basis. it used to take 15-20mins until a week back.

What's the source of the slow copy activity?

Azure Table storage

What's the sink of the slow copy activity?

Azure data lake storage gen2

Which type of integration runtime are you using?

Azure IR

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2024-09-19T08:03:54.6633333+00:00

Mayuri Shah ,

Just following up to see if the below answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'
Mayuri Shah 0 Reputation points

2024-09-19T23:25:19.5866667+00:00

more details: Data size has been doubled, hence the issue. what can we do to reduce copy activity time with double data size?
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2024-10-03T05:41:48.32+00:00

Mayuri Shah ,

If the volume has doubled, it will certainly impact the execution time . To reduce the execution time , you can try increasing the DOP in copy activity so that it increases parallel processing. But make sure not to increase it exponentially. Try with small values .

Other option in case you are using self hosted IR, is to use distribute the load into multiple nodes by installing IR in multiple VMs. You can leverage 4 nodes at max.

References: Performance tuning steps in copy activity in ADF

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

2 answers

Your answer

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2024-09-19T08:03:54.6633333+00:00

Mayuri Shah ,

Just following up to see if the below answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'
Mayuri Shah 0 Reputation points

2024-09-19T23:25:19.5866667+00:00

more details: Data size has been doubled, hence the issue. what can we do to reduce copy activity time with double data size?
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2024-10-03T05:41:48.32+00:00

Mayuri Shah ,

If the volume has doubled, it will certainly impact the execution time . To reduce the execution time , you can try increasing the DOP in copy activity so that it increases parallel processing. But make sure not to increase it exponentially. Try with small values .

Other option in case you are using self hosted IR, is to use distribute the load into multiple nodes by installing IR in multiple VMs. You can leverage 4 nodes at max.

References: Performance tuning steps in copy activity in ADF

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Answer 1

Verify if there are any performance bottlenecks on the Azure Table Storage side, such as increased data volume or throttling limits being hit. Check if the storage account is under stress or if there have been changes to the access patterns.

Assess if there has been an increase in the volume of data being copied recently. A larger dataset will naturally take longer to transfer. Compare the data volumes from previous days to identify any spikes in the size of the source data.

Ensure that the integration runtime is scaling appropriately. If the IR is overloaded or under-provisioned, it might slow down the data transfer process. Check the performance metrics of the IR, such as CPU usage, memory consumption, and network bandwidth.

You may need to upscale it to Azure Support if needed.

Answer 2

Hi Mayuri Shah ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

I understand that you are facing performance issues while trying to copy data from Az table storage to ADLS gen2 using copy activity via Azure IR .

As you mentioned, comparatively, earlier last week it was taking lesser time. Could you please check if there is significant increase in the volume of data getting copied as compared to last week? Kindly check the copy activity output log and see the datacopied size and compare .

Also , is there any configuration related changes that were made in the recent times like changing the DIU, or DOP ? Kindly check.

Additionally, what is the region in which ADF is provisioned, I would check if there is any ongoing outage or any service health impact in that region in recent times.

Looking forward to your response. Thankyou

Share via

Copy Activity very slow for Azure data factory

2 answers

Your answer