Azure Data factory pipeline stuck at a step to load data for hours...sometimes, it is success in 4-5 mins.

Question

We have a scheduled run on ADF pipeline increment loading data from Blob storage to Azure Postgres database. table has 241 columns. sometimes, it runs successfully in 4-5 mins and other time it keeps running for hours...We can successfully run it manually debug it and almost every time it completes in 4-5 mins. Please help and advise how to fix it. It is not giving any error message as it keeps running for long time.

Answer 1

One possible cause of the issue could be hitting concurrency limits. To check this, verify that there are no ongoing old pipeline runs if your pipeline has a concurrency policy. You can do this by going to the ADF authoring canvas, selecting your pipeline, and checking if a concurrency property is assigned. If it is, switch to the Monitoring view and ensure there are no runs in progress from the past 45 days. If there are, cancel them to allow the new pipeline run to start.

Another potential issue could be monitoring limits. Additionally, your run might be affected by transient network issues, credential failures, service outages, etc. Azure Data Factory has an internal recovery process that monitors runs and restarts them if it detects a problem. This process runs every hour. If your run has been stuck for more than an hour, create a support case.

For more detailed information, refer to the Azure Data Factory documentation on handling pipeline statuses that are queued or stuck for a long time.

https://learn.microsoft.com/en-us/azure/data-factory/monitor-visually#rerun-pipelines-and-activities

https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide#pipeline-status-is-queued-or-stuck-for-a-long-time