Azure Data factory V2 pipeline trigger stuck with In-progress status?

Nikunj Patel 186 Reputation points
2020-07-06T08:58:50.75+00:00

Actually, we have one pipeline with 30 different schedules based on different pipeline parameter. Some of schedule executed every 10 min and some of executed once in a day. After setup the pipeline it running fine for one day. But after that suddenly all pipeline trigger run stuck in In-progress status and I am not able to find any root cause. So can anyone help me How we can find the root cause or logs. Also how we resolve the issue? approx more than 5000 trigger schedule with In-progress status. Please anyone provide solution asap.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,139 questions
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,061 Reputation points
    2020-07-07T20:37:20.13+00:00

    Thank you for the information @NikunjPatel-2604 .

    Given that all these pipelines are In-Progress, I suspect some resource is not responding, and the activities are left waiting for answer. They will wait until the Timeout property is reached.
    Since you are using a database, the next step in finding root cause, is to look at whether the database is actually executing the requests. You are using lookups and stored procs. First let us find out which step the pipeline is stuck on.
    Go to the Data Factory Monitoring section, and find one of the stuck pipeline runs. Click on the name and it should show you the status of the activities. From this you should be able to determine whether it is lookup or stored proc that is stuck.
    Get with your database person and check the logs in the database as to what happened to the request database side. Did the request execute? Did the request get stuck? Or did the request never make it into the database?

    For the immediate situation, I recommend you make 2 changes to the pipeline (for now).

    • Set the pipeline concurrency to 100, or some number you are comfortable with. If no number is used, more and more will be triggered. The concurrency is total for the pipeline, not per trigger, but until the situation is stabilized, it doesn't matter.
    • Set the Timeout property of the Stored Proc and Lookup activities to an hour or some value you are comfortable with. The default value is 7 days, meaning that the activity waits 7 days before giving up and failing. Reducing this value will cause new executions to expire sooner.

    To stop all the currently executing runs, we can use Powershell (or other methods) to get the list of pipeline runs and stop each.

    Get-AzDataFactoryV2PipelineRun
    stop-azdatafactoryv2pipelinerun

    Get all the runs of the pipeline in a time window: $list =Get-AzDataFactoryV2PipelineRun -ResourceGroupName <RgName> -DataFactoryName <FactoryName> -PipelineName <ThatPipelinename> -LastUpdatedAfter <startedDate> -LastUpdatedBefore <nowDate>
    Get the RunId's of those that are in-Progress: $toStop = $list.ForEach({ if( $_.Status -eq "InProgress") { $_.RunId }} )
    Stop them by RunId: $toStop.ForEach({ Stop-AzDataFactoryV2PipelineRun -ResourceGroupName <RgName> -DataFactoryName <FactoryName> -PipelineRunId $_ })


0 additional answers

Sort by: Most helpful