DataFactory: Activites stuck in the queued state

Maryia T 1 Reputation point
2021-09-13T11:22:04.667+00:00

We have a number of DataFactory pipelines using Self-Hosted Integration Runtime for Looking up data from Azure SQL/SQL Servers and copying data from various sources based on the lookups response. Normally, lookups take seconds (1-5 seconds, a small SQL query on a table of <1000 records), copy duration usually varies from minutes to a couple of hours. This setup has been operating for over a year now with the same amount of jobs/data.

Recently, the following issue has been arising on an almost daily basis: all triggered activities are marked as 'Queued', whereas there are 0 activities in progress and IR CPU utilization is 0%:

131595-image.png

It doesn't help when the queued pipelines are cancelled, the queue length usually stays the same. This state can last for days (48 hours of weekend at least) basically blocking our normal operations. Restart of the IR helps - after it some of the queued activites transfer into "in progress" and everything starts operating as normal.

Obviously, manual restart multiple times a day is not an option. We would also like to understand why it's happening and how to prevent this kind of "deadlock" happening in the future. Thanks in advance.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,893 questions
{count} votes