ADF copy activities queue for a long time before they start.

Question

ADF copy activities queue for a long time before they start. It happens with the Autoresolveintegration runtime (West Europe). Is there an outage in West Europe? Or how can this be prevented? Every activity queues for a minimum of 4 minutes.

Answer 1

@Nils Wagenaar

Welcome to Microsoft Q&A platform and thanks for posting your question here.

I see there are no reported outages in the West Europe region, there might be various reasons why ADF copy activities are taking a long time to start.

Here are some steps you can take to troubleshoot and resolve the issue:

Check if the integration runtime is experiencing performance issues. You can do this by going to the Azure portal, selecting the integration runtime, and checking the performance metrics. If the integration runtime is experiencing performance issues, you may need to upgrade to increase the node to balance the activities.

Check if your pipeline has a concurrency property assigned to it. If it does, go to the Monitoring view, and make sure there's nothing in the past 45 days that's in progress. If there is something in progress, you can cancel it, and the new pipeline run should start.

It is possible that your run was impacted by a transient network issue, credential failures, services outages, etc. If this happens, Azure Data Factory has an internal recovery process that monitors all the runs and starts them when it notices something went wrong. You can rerun pipelines and activities as described here: his process happens every one hour, so if your run is stuck for more than an hour, create a support case.

Break up copy activities into smaller chunks If you are copying a large amount of data, consider breaking up your copy activities into smaller chunks to reduce the time it takes for the activities to start.

If you are experiencing performance issues with your data stores, consider optimizing their performance to ensure that copy activities can run smoothly.

Reference

https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance

https://learn.microsoft.com/en-us/azure/data-factory/monitor-visually#rerun-pipelines-and-activities.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Hello @Nils Wagenaar

There could be several reasons why ADF copy activities are queuing for a long time before they start. One possible reason could be that the Autoresolveintegration runtime in West Europe is experiencing high demand and is unable to keep up with the workload.

Another possible reason could be that there are performance issues with the copy activity itself. To troubleshoot this issue, you can try the following steps:

1. Check the status of the Autoresolveintegration runtime in West Europe to see if there are any known issues or outages. You can do this by going to the Azure portal and navigating to the Integration Runtimes section of your Data Factory.
2. Check the performance of the copy activity itself. You can do this by reviewing the troubleshooting steps outlined in the Copy Performance Improvement article. This article provides guidance on how to optimize copy activity performance for different data sources.
3. If the issue persists, you can try scaling out your Autoresolveintegration runtime by adding more nodes to balance the workload. You can also try using a different integration runtime in a different region to see if that improves performance.

I hope this helps! Let me know if you have any further questions.