ADF activity to check for file existence in ADLS and move to subsequent activity on success else it should stop, However pipeline should not get failure when file doesn't exist

Question

Hi All,

Good Day!!

ADF activity to check for file existence in ADLS and move to subsequent activity on success else it should stop, However pipeline should not get failure when file doesn't exist but it should get stop and other subsequent activities should not execute.71201-tablelog-2.txt

I already tried with validation activity in adf, it is working as expected when the file exist in adls, However pipeline is getting failure when files doesn't exist.

Answer 1

Hi @Abdulla Mahammad Khan Dawood ,

Thanks for reaching out. In addition to @Nasreen Akter 's inputs please find below my insights.

The success/failure of a pipeline depends upon the dependencies attached to a failing activity. The shortest way to explain this is "If there is a success path (Green path), and that path is not taken, then the pipeline status is failure."

If an activity fails, and there are no dependent activities following it, then the pipeline fails. (In other words, if the last activity in a pipeline fails, the pipeline fails.)

If an activity fails, and it is followed only by another activity connected by a green on-success path, then the pipeline fails.

If an activity fails, and it is followed only by another activity connected by a red on-failure path, then the pipeline succeeds.

If an activity fails, and it is followed only by another activity connected by a blue on-completion path, then the pipeline succeeds.

If an activity fails, and it is followed by two activities, one connected by a red on-failure path, the other connected by a green on-success path, then the pipeline fails.

If an activity fails, and it is followed by two activities, one connected by a red on-failure path, the other connected by a blue on-completion path, then the pipeline succeeds.

In your case, you can use On-completion dependency path after validation activity and have a if condition activity to check if the file exists or not and if exists take true path and continue the flow, else it will fall to false path and end the pipeline run without failure. Please see below GIF

Here are couple of other threads on similar requirement and hope those will help you understand the use case of dependency paths.

1. Understanding Pipeline Failures and Error Handling
2. Azure data factory pipeline run status "succeeded" when failed activity is followed by other activity (on failure / on completion)
3. ADF - Can validation activity timeout be suppressed and not be shown as a failure ?

Hope this clarifies.

----------

Thank you
Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

Announcement : Want to participate in Azure Data Factory hackathon'21? Come hack with us :) - https://aka.ms/adfhack21 (Note: Submission Deadline 24th Feb'21)

Answer 2

Hi @Abdulla Mahammad Khan Dawood ,

Please use GetMetaData instead. In the GetMetadata, please select Field list --> Exists and in the if condition check

@activity('Get Metadata Description').output.exists  

Please see the screenshots for details.Thanks!

----------

If the above response is helpful, please accept as answer and up-vote it. Thanks!