Copy Activity very slow for Azure data factory

Question

Hi All,

we are having issue where copy activity from Azure Table storage to data factory is taking over 50mins on daily basis. it used to take 15-20mins until a week back.

What's the source of the slow copy activity?

Azure Table storage

What's the sink of the slow copy activity?

Azure data lake storage gen2

Which type of integration runtime are you using?

Azure IR

Answer 1

Hi Mayuri Shah ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

I understand that you are facing performance issues while trying to copy data from Az table storage to ADLS gen2 using copy activity via Azure IR .

As you mentioned, comparatively, earlier last week it was taking lesser time. Could you please check if there is significant increase in the volume of data getting copied as compared to last week? Kindly check the copy activity output log and see the datacopied size and compare .

Also , is there any configuration related changes that were made in the recent times like changing the DIU, or DOP ? Kindly check.

Additionally, what is the region in which ADF is provisioned, I would check if there is any ongoing outage or any service health impact in that region in recent times.

Looking forward to your response. Thankyou

Answer 2

Verify if there are any performance bottlenecks on the Azure Table Storage side, such as increased data volume or throttling limits being hit. Check if the storage account is under stress or if there have been changes to the access patterns.

• Assess if there has been an increase in the volume of data being copied recently. A larger dataset will naturally take longer to transfer. Compare the data volumes from previous days to identify any spikes in the size of the source data.

Ensure that the integration runtime is scaling appropriately. If the IR is overloaded or under-provisioned, it might slow down the data transfer process. Check the performance metrics of the IR, such as CPU usage, memory consumption, and network bandwidth.

You may need to upscale it to Azure Support if needed.