ADF - Restart Pipeline - Start from Failure

Kothai Ramanathan 941 Microsoft Employee

On failure of a pipeline, I am trying to rerun the pipeline, by providing the run id and startFromFailure as true.

Now in my pipeline, I had a foreach activity which processed 3 files - out of which 2 files were successful and 1 file failed. When I run with start from failure, I expected the foreach to start processing only from the 1 file which failed. But it it now trying to pick all the three files and process it.

The two files which were successful in the earlier run are not longer there (as it is copy and delete). Now I get failure for the already successfully processed files.

How can I make the pipeline run only for the failed file and not pick the earlier two files which were successful.

Any pointers will be helpful, thanks.

KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2020-09-25T05:10:23.023+00:00

Hi @Kothai Ramanathan ,

Thanks for your query. Could you please elaborate on this 'The two files which were successful in the earlier run are not longer there (as it is copy and delete). Now I get failure for the already successfully processed files.' ? My understanding is that even though those two files were already processed, when you try to rerun from the point of failure, pipeline is looking for those files which were already processed and throwing error. If that is case could you please share complete error message and pipeline runID & activity runID? Also are you using Delete activity to perform deletion of files after copy or some other logic?

Looking forward to your response.

Thank you
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2020-09-28T22:09:59.01+00:00

Hi @Kothai Ramanathan ,

Just checking to see if you have got a chance to see my previous comment, if so could you please provide few additional details as called out in my previous comment?

Thank you

Share via

ADF - Restart Pipeline - Start from Failure

Your answer