ADF - Can validation activity timeout be suppressed and not be shown as a failure ?

Question

ADF - Can validation activity timeout be suppressed and not be shown as a failure ?

Kothai Ramanathan 946 Microsoft Employee

I have a pipeline which process based on the availability of files in a folder. I have a validation activity, which validates whether files are available or not. If files are not available, nothing needs to be done.

In this pipeline, the run is fine. I get a timeout, and the 'Do nothing' activity succeeds. However, now I am calling this pipeline from a parent pipeline with the executepipeline activity. Now when files are not found, the parent pipeline's executepipeline activity is failing with the message : "Operation on target Validate if files exist failed". I do not want the parent pipeline to fail when there are no files found, as this is not actually a failure.

How can I do this ? Any pointers will be helpful, thanks

SantoshKesava-4601 6 Reputation points

2022-10-20T14:37:32.287+00:00

though is very old post , but posting my work around to let others know.

File Validation task will be timed out when it did not find the file within the time mentioned. The nature of the ADF pipeline is if any at least one task that did not run inside the pipeline , the pipeline will automatically goes to error state.

In file validation case also , after the timeout the other tasks connected to the success port couldn't be run and therefore the error.
To overcome this , there's need to be a an IF task connected to "Completion" port ( not "Success" port ) and in the IF condition false can send an email saying that the file is missing and True will have all the necessary tasks.

I hope this helps.

Thank you

1 answer

Your answer

SantoshKesava-4601 6 Reputation points

2022-10-20T14:37:32.287+00:00

though is very old post , but posting my work around to let others know.

File Validation task will be timed out when it did not find the file within the time mentioned. The nature of the ADF pipeline is if any at least one task that did not run inside the pipeline , the pipeline will automatically goes to error state.

In file validation case also , after the timeout the other tasks connected to the success port couldn't be run and therefore the error.
To overcome this , there's need to be a an IF task connected to "Completion" port ( not "Success" port ) and in the IF condition false can send an email saying that the file is missing and True will have all the necessary tasks.

I hope this helps.

Thank you

Answer 1

HarithaMaddi-MSFT 10,146

Hi @Kothai Ramanathan ,

Thanks for posting the question.

When we have success and failure flows and if any activities under success flow are skipped, the pipeline shows as "Failed". If the child pipeline fails, by design, the parent pipeline also fails. This scenario can be avoided by using skip connector between every success flow activity and failure flow activity indicating to ADF that they can be skipped. In this case when files are not found as well, parent pipeline would not fail. Kindly check the below snap for the same.

Couple of workarounds to avoid this scenario would be as below.

If it is activity other than 'set variable', 'If Condition' activity can be used to check the status of the activity with following example for "Copy Activity". However, this has limitation that we cannot use 'Foreach activity' inside IF activity. Ex: activity('Copy_Activity1').output.executionDetails[0].status
To have parent pipeline trigger the actual pipeline as child pipeline using "Execute Pipeline" activity and handle error logging in parent pipeline, but this has limitation if specific failed activity information is needed from Child Pipeline that caused the error.

There is a request for adding OR condition between success and failure flows in ADF similar to SSIS. Please consider to upvote the idea if you think its good feature to have. All the feedback you share, is closely monitored by the Product team and implemented in future releases.

35616982-allow-choosing-logical-and-or-logical-or-in-activi

Hope this helps! Please let me know for more questions and I will be glad to assist further.

Kothai Ramanathan 946 Reputation points Microsoft Employee

2020-08-24T04:18:52.183+00:00

Hi, in my case the child pipeline succeeded when there were no files. Only the parent failed. Please advice.
Kothai Ramanathan 946 Reputation points Microsoft Employee

2020-08-24T04:31:34.537+00:00

My above comment is not right. The child pipeline is failing. After the validation activity, I have 6 activities. Should all of them have a skip activity for it to succeed ? If yes, is there a better way to handle this ? Please let me know.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-24T06:17:43.15+00:00

Hi @Kothai Ramanathan ,

Thanks. I have added couple of workarounds for this in the above answer. Also, posted an open feedback item related to this feature, kindly suggest if this helps. Please let us know for further queries and we will be glad to assist.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-26T05:21:18.097+00:00

Hi @Kothai Ramanathan ,

Checking to see if above answer clarified the question. Please let us know for further queries and we will be glad to assist.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-27T06:22:58.953+00:00

Hi @Kothai Ramanathan ,

We still have not heard back from you. Please let us know for further queries and we will be glad to assist.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Kothai Ramanathan 946 Reputation points Microsoft Employee

2020-08-28T09:11:21.417+00:00

Thank you for the details. However, I did not understand the below comment:
"If it is activity other than 'set variable', 'If Condition' activity can be used to check the status of the activity with following example for "Copy Activity". However, this has limitation that we cannot use 'Foreach activity' inside IF activity. Ex: activity('Copy_Activity1').output.executionDetails[0].status".

I basically do not want the validation activity to not fail when no files are found. In my pipeline, the validation activity is the first activity, and there are 6 more activities following this. Please suggest an optimal way to do this, thanks
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-28T12:45:38.367+00:00

Hi @Kothai Ramanathan ,

Thanks for checking back on it. I have made the below GIF to explain the workaround using 'IF Condition' activity as below. This will ensure the pipeline is successful even in case of no files but foreach cannot be used inside "IF Condition" hence I recommend to use child pipeline for success activity in your scenario with all of them moved to a different pipeline. Hope this helps!

Please let us know for further queries and we will be glad to assist.
Kothai Ramanathan 946 Reputation points Microsoft Employee

2020-08-31T03:58:31.063+00:00

Hi... not able to understand the workarounds. My scenario is simple... I have validation as the first activity that checks if files are present in a folder. Need to proceed only if files are present and not throw an error when no files are present. After the validation activity, there are 6 more activities (getmetadata, foreach, and delete activities). Please suggest the best and optimal way to handle this scenario. Thanks in advance.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-31T08:37:02.287+00:00
Hi @Kothai Ramanathan ,

Apologies that the workaround is not properly conveyed to you. Please find below another GIF that might make it clear to implement it for your scenario.

Couple of points :

As per ADF design, it fails the pipeline when there are both failure and success flows and when any activity in success flow is skipped.

Workarounds suggested will need certain amount of re-work and can be chosen according to business requirements.

If workarounds are tedious for your requirement, please upvote the Feedback item shared above and ADF product team will prioritize and implement it.

If below GIF does not help, kindly share a mail to AzCommunity[at]microsoft[dot]com with below details and we will connect with you

Subject: "Attn: Haritha - ADF - Can validation activity timeout be suppressed and not be shown as a failure ?"
Thread URL:
SantoshKesava-4601 6 Reputation points

2022-10-19T20:40:34.243+00:00

Hi @HarithaMaddi-MSFT , your comment has given me a hope to overcome this problem. In my situation I have an independent single pipeline ( not parent not child ) with the same scenario. Shall I use if block after validation task ? Please suggest. Thank you

Share via

ADF - Can validation activity timeout be suppressed and not be shown as a failure ?

1 answer

Your answer