Pipelines suddenly stops executing

Mathias Bundgaard 5

The duration of the pipeline "Extract_mgmtsys-Expenses" (15m 26s) does not match the sum of its sub-activities' durations (6m 30s). This discrepancy arises because there is a delay between the completion of one activity and the start of the next, despite the previous activity completing successfully. This indicates that the activities are not queued immediately after each other. How can this issue be resolved?

Total duration 15m 26s

Sub activites (Summed up): 6m 30s User's image

Suba Balaji 11,206 Reputation points

2024-03-21T03:24:35.39+00:00

Hi, the duration we see on the output window is just a fraction of whole process. Executing activities involve some pre-processing and post-processing activities like establishing connectivity, and releasing the same, and logging if any, doing consistency checks, etc.

Please let us know when you see the pipeline stops execution suddenly as above screenshot shows that pipeline is running fine. So please share details on when you encounter issues with respect to the overall run time of pipeline.
Smaran Thoomu 12,620 Reputation points Microsoft Vendor

2024-03-22T07:58:49.7033333+00:00

@Mathias Bundgaard We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Mathias Bundgaard 5 Reputation points

2024-03-22T08:29:44.65+00:00

Thank you for your reply.

The problem i am experiencing is inconsistencies in pipeline performance in Azure Data Factory.

The same pipeline as previous post (See pictures), without any modifications or changes to the environment, sometimes executes significantly faster. This discrepancy occurs despite the activities within the pipeline remaining the same. What causes such variable performance, and how can it be resolved? I'm seeking an understanding of the mechanisms behind pipeline execution in Azure Data Factory to formulate a solution.

This inconsistency is not isolated to a single example but observed across various instances. Typically, I trigger a master pipeline that initiates 20-25 sub-pipelines simultaneously.

See the same pipeline with the same activities executed.
Suba Balaji 11,206 Reputation points

2024-03-23T05:02:06.2166667+00:00
Hi Mathias,

It may be related to the difference in volume of data being handled from time to time. Kindly check the amount of rows being processed every time to get an idea.

There's no data flow involved here. All the activities are pretty straight forward. So it could be the slowness in the region specifically. Please let us know in which region do you encounter this issue.

You say that you trigger from master pipeline. How much time it takes for the master pipeline to trigger the child pipeline? Can you please post screenshot of the master pipeline runtimes
Smaran Thoomu 12,620 Reputation points Microsoft Vendor

2024-03-26T08:29:12.7433333+00:00

@Mathias Bundgaard We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Pipelines suddenly stops executing