ADF copy activities queue for a long time before they start.

Nils Wagenaar 0 Reputation points
2024-03-06T12:35:21.6866667+00:00

ADF copy activities queue for a long time before they start. It happens with the Autoresolveintegration runtime (West Europe). Is there an outage in West Europe? Or how can this be prevented? Every activity queues for a minimum of 4 minutes.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,456 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Harishga 3,170 Reputation points Microsoft Vendor
    2024-03-07T05:19:15.8533333+00:00

    @Nils Wagenaar

    Welcome to Microsoft Q&A platform and thanks for posting your question here.

    I see there are no reported outages in the West Europe region, there might be various reasons why ADF copy activities are taking a long time to start.

    Here are some steps you can take to troubleshoot and resolve the issue:

    Check if the integration runtime is experiencing performance issues. You can do this by going to the Azure portal, selecting the integration runtime, and checking the performance metrics. If the integration runtime is experiencing performance issues, you may need to upgrade to increase the node to balance the activities.

    Check if your pipeline has a concurrency property assigned to it. If it does, go to the Monitoring view, and make sure there's nothing in the past 45 days that's in progress. If there is something in progress, you can cancel it, and the new pipeline run should start.

    It is possible that your run was impacted by a transient network issue, credential failures, services outages, etc. If this happens, Azure Data Factory has an internal recovery process that monitors all the runs and starts them when it notices something went wrong. You can rerun pipelines and activities as described here: his process happens every one hour, so if your run is stuck for more than an hour, create a support case.

    Break up copy activities into smaller chunks If you are copying a large amount of data, consider breaking up your copy activities into smaller chunks to reduce the time it takes for the activities to start.

    If you are experiencing performance issues with your data stores, consider optimizing their performance to ensure that copy activities can run smoothly.

    Reference

    https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance

    https://learn.microsoft.com/en-us/azure/data-factory/monitor-visually#rerun-pipelines-and-activities.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments