Azure Data Factory Until loop slow to end/terminate

Geezer 16

We have an until loop in a ADFv2 pipeline.

The time it takes to stop/terminate once the expression condition is met seems to corrolate between the length of time the until loop takes to completes its activities.

This particular until loop performs alot of activites and can take anywhere between 90-120 mins to complete. So it takes almost as long to end/terminate (break out of the loop).

If I "hack" it so that it only performs a handful of activities it will quickly end and break once it's finished and the expression to terminate is met.

It's like a spinning wheel that keeps spinning even after the power is turned off. The momentum that was built up while connected takes a while to slow down and eventually stop.

Is this a known issue, how can I troubleshoot the exact cause here or fix it?

ShaikMaheer-MSFT 38,126 Reputation points Microsoft Employee

2022-10-11T13:55:44.123+00:00

Hi @Geezer ,

Thank you for posting query in Microsoft Q&A Platform.

Looks strange, Are we sure when until condition is becoming true there is not activity under in progress or queue?

I am suspecting there may be some internal activity of loop still running or yet to complete, hence its taking time to break the loop after condition meet. Kindly recheck and confirm, if possible, with screenshots and details. So that it will be more understandable to get idea about what exact causing this. Thank you.
Geezer 16 Reputation points

2022-10-12T09:43:46.607+00:00

Hi @ShaikMaheer-MSFT

There is no activity in a queue or in progress state.

These are screenshots from today.

This shows the pipeline in which the UNTIL loop resides in.
Started at 2022.11.10 11.01.56pm last night.

This is the UNTIL loop.

The expression is

Expanded out

The Action variable is set by the Determine Action activity.

Here’s a screenshot of the most recent rows from the Activity run log. There are typically 4k plus rows here.

The Determine Action step at 07.06am set the Action variable to “Quit”.

The last step shown within the UNTIL activity was executed at 07.07am this morning. It is now 09.40am. The UNTIL has still not broken out.
ShaikMaheer-MSFT 38,126 Reputation points Microsoft Employee

2022-10-12T10:17:00.07+00:00

Hi @Geezer , Thank you for sharing details with screenshots. Everything looks good to me. Only one thing I am suspecting is Stored procedure activity. Its inside loop and not connected with any other activities. That means it runs in parallel for every iteration. Can we try to add that Stored procedure activity with green line and then see how it behaves? Since its not connected with green lines I suspect that activity may be running in background. I may be wrong too. But to make sure we can try to connect that activity also with green line and see the execution.
Geezer 16 Reputation points

2022-10-12T11:44:10.027+00:00

Hi @ShaikMaheer-MSFT

That stored procedure activity is on it's own as it's set to Poll/Retry every 30 seconds, independant of what else is going on in the until loop. Adding a green line between the Take Action switch and the stored procedure activity will negate the polling/retry property. That isn't an option.

Since I took the previous snapshots the until loop has finished.
There is a gap of over 3 hours (from 07.07.52am to 10.16.34am) between the final activity in the until loop finishing and the start of the first activity outside of it.

Why did it take over 3 hours?

Like I mentioned in the initial post, if this same until loop only executed a handful of activies instead of the 4k plus, it breaks quickly.
ShaikMaheer-MSFT 38,126 Reputation points Microsoft Employee

2022-10-12T11:52:00.57+00:00

Hi @Geezer , Looks very strange. let me check with some internal team if they have any insights on this. Also, you can consider having a support case too for deeper investigation and understand what is happening exactly here.
Geezer 16 Reputation points

2022-10-13T14:50:31.133+00:00

Hi @ShaikMaheer-MSFT

Have you heard back from the internal team at all?
ShaikMaheer-MSFT 38,126 Reputation points Microsoft Employee

2022-10-14T04:47:31.19+00:00

Hi @Geezer , I tried to reach some internal teams if any insights on this. Not heard back anything yet. I will be updating thread once I get some insights. You can also consider raising support case too for deeper investigation and resolution on this. Below is the link with steps have support case. Thank you.
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request
Ravindra Chippada 1 Reputation point

2022-11-16T13:13:13.887+00:00

@ShaikMaheer-MSFT
We're facing similar issue. Until loop taking extra time to terminate even after condition met. Our activities inside Until are Web activity which polls for a status, then it'll update a variable called 'status'. Then there is a if condition to validate status variable, if its 'failed' or 'completed' nothing happens else there is a wait time of 30 secs.
I could see variable 'status' is getting updated to 'Completed' and eventually the condition defined in Until is met, but still its running for 2-3 mins more.
Florent Pousserot 6 Reputation points

2022-12-23T15:06:07.98+00:00

i open this case : 2212230050000981

the kudo query

ADFActivityRun
| where PipelineRunId == 'xxxxxxxxxxx'
| where _ResourceId == "/subscriptions/xxxxx/resourcegroups/rxxxx/providers/microsoft.datafactory/factories/xxxxxx"
| where Status == 'Succeeded'
| sort by TimeGenerated desc
| project TimeGenerated, Start, End, TimeTotal_sec = datetime_diff('Second', End, Start), Delay_sec = datetime_diff('Second', prev(End), End ), OperationName, Status
| extend rn=row_number()
| limit 11
Henk Standhardt 1 Reputation point

2023-01-02T13:59:40.603+00:00

We experience the same behavior. Should we file a support ticket?
Viguro 1 Reputation point

2023-01-12T05:03:49.3066667+00:00

@Ravindra Chippada We are also facing the same issue :

A web activity polls for a status, then update a variable. If this variable is failed or completed nothing happens, else it wait for 10s.

Will the last activity if the Until loop ends at 10:20:29, the until loop ends 20 minutes later. The total execution time of our pipeline is 2h17.

6 answers

Viguro 1 Reputation point

2023-08-04T07:02:27.43+00:00

Duplicate comment
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Azure Data Factory Until loop slow to end/terminate

6 answers