ADF Pipeline-Loop starts over again when the inner activity fails

Question

ADF Pipeline-Loop starts over again when the inner activity fails

Ramamoorthy, Kesavan 61

Hi Experts,

The picture below shows the activities inside Loop in my ADF pipeline wherein we have a copy activity that fetches data from an api using http connector.

Whenever the copy activity fails due to some reason in the api, the loop goes back and starts from the 1st step. For example, when loop starts with the first iteration and when copy activity fails, it goes back to the first step i.e. "Get Access Token2" and start running from there. Is this a normal behavior? Sorry I am fairly new to Azure.

Is there any way that we stop the loop as soon as the copy activity fails.? Basically it should not go back to the first step of the loop when the copy activity fails.

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-11T12:36:15.753+00:00

Hello @Ramamoorthy, Kesavan ,

Just checking in to see if the below answer provided by anonymous userChaudhari helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Ramamoorthy, Kesavan 61 Reputation points

2021-03-11T14:22:46.333+00:00

Hi @PRADEEPCHEEKATLA ,

I am implementing that approach at the moment and waiting to get the required access in order to assign the contributor role to the ADF object as per Andy's direction in that blog. Will keep you posted on it if that solves the issue.

I have another question. Sometimes some of the activities remain in Queue state for some time and then only it moves to Inprogess state. For example, sometimes the copy activity takes 25seconds out of which queue time is 23secs and the transfer time is just 2secs. So what can be done to avoid/resolve this queue timing? The standard 'AutoResolveIR' is being used and the region is West Europe. Also could you pls let me know where/how can I see how much is the queue time for activities like Lookup because the 'Details' pane is coming up only for Copy activity in the run log.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-12T07:43:44.76+00:00

Hello @Ramamoorthy, Kesavan ,

Initially, the orchestration ADF puts the job in a queue and it gets "queue time" until the infrastructure picks it up and start the processing part.

As per the SLA for Data Factory, We guarantee that at least 99.9% of the time, all activity runs will initiate within 4 minutes of their scheduled execution times.

For more details, refer SO thread which addressing similar issue.

1 answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-11T12:36:15.753+00:00

Hello @Ramamoorthy, Kesavan ,

Just checking in to see if the below answer provided by anonymous userChaudhari helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Ramamoorthy, Kesavan 61 Reputation points

2021-03-11T14:22:46.333+00:00

Hi @PRADEEPCHEEKATLA ,

I am implementing that approach at the moment and waiting to get the required access in order to assign the contributor role to the ADF object as per Andy's direction in that blog. Will keep you posted on it if that solves the issue.

I have another question. Sometimes some of the activities remain in Queue state for some time and then only it moves to Inprogess state. For example, sometimes the copy activity takes 25seconds out of which queue time is 23secs and the transfer time is just 2secs. So what can be done to avoid/resolve this queue timing? The standard 'AutoResolveIR' is being used and the region is West Europe. Also could you pls let me know where/how can I see how much is the queue time for activities like Lookup because the 'Details' pane is coming up only for Copy activity in the run log.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-03-12T07:43:44.76+00:00

Hello @Ramamoorthy, Kesavan ,

Initially, the orchestration ADF puts the job in a queue and it gets "queue time" until the infrastructure picks it up and start the processing part.

As per the SLA for Data Factory, We guarantee that at least 99.9% of the time, all activity runs will initiate within 4 minutes of their scheduled execution times.

For more details, refer SO thread which addressing similar issue.

Answer 1

Yes, this is expected behavior. Loop starts and if some activity within loop fails it iterates for next loop and run all inner activities again.

You will have to customize the solution with some flag logic to break the loop.

Refer below blog for the logic -
https://andyleonard.blog/2020/06/one-way-to-break-out-of-an-azure-data-factory-foreach-activity/

You can also upvote existing feedback here -
https://feedback.azure.com/forums/270578-data-factory/suggestions/39673909-foreach-activity-allow-break

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

Share via

ADF Pipeline-Loop starts over again when the inner activity fails

1 answer

Your answer