How to implement logic if one of any five different boolean variable are true...

JasonW-5564 161 Reputation points
2021-09-30T20:11:22.63+00:00

I have a daily ELT process that picks up some file from blob storage. This process has five required files that need to be present for it to proceed. This is the part of the logic I am trying to implement.

I already have the logic that checks to see if 5 required files are present or not and sets a boolean type value for each if they are missing (example: File1_is_missing = true if the file is not present in BLOB storage). I want to send an email and fail my pipeline if any of the five boolean variables are true. How can implement this? All I can think of is some kind of big nasty nested IF activity and I am not even sure how to do it...

NOTE: I want to evaluate all five variables before sending the email because I only want to send one email and have logic in the body of that email that lists any of the missing files. Hope that makes sense!!! Thanks for any help/ideas!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. svijay-MSFT 5,206 Reputation points Microsoft Employee
    2021-10-01T12:55:06.327+00:00

    Hello @JasonW-5564 ,

    Thanks for the question and using MS Q&A platform.
    I see you already have 5 Boolean variables.

    So, you could do the below :

    From the output where you set the variable to true (For instance, File1_is_missing is true)- You could have an IF condition

    This IF condition has one activity which adds/appends 'Filename' to an Array Variable - when value is true - using the Append to Variable activity - Let's assume this to be MissingFilesList. You need to declare this as an array.

    137012-image.png

    Refer the AppendFile1,AppendFile2 in the below screenshot.

    136950-image.png

    On every true, it will add the filename to the variable MissingFilesList

    Now, there is another IF (If Condition 3) which checks the length of MissingFilesList variable, if it is 0 - it means that there are no files missing. Else, there is a missing a file. You now have the list of missing files in the variable MissingFilesList

    @equals(length(variables('MissingFiles2')),0)  
    

    If you want to proceed you can define the activities in the true section of this condition.

    Alternative

    The other approach I thought of is below

    136926-image.png

    For every non existing File, the filename is appended to an array variable - MissingFiles.
    The below is the expression for AppendMissingFile1.

    @if(activity('GetFile1').output.exists,'','File1')  
    

    You will repeat the process for all the five files.

    Note: You will be appending empty strings ''when doing the above action to the array.

    So, you will have to eliminate the same. That is being taken care the in the GettheMissingFiles (Set the Variable activity) - MissingFileswithoutBlank

    @intersection(createArray('File1','File2'),variables('MissingFiles'))  
    

    The output of this action will contain the name of all missing files in an Array variable (MissingFileswithoutBlank)
    Like the above explained step, you can get the length of the collection (MissingFileswithoutBlank) - if it is 0 - you can proceed with the necessary activities, if not you can do the failure activities in the false activities.

    **On Failure - when files are missing **

    You can execute the below activity false section of last the IF condition (If condition 3 & 1 ) in above screenshots.

    For email - You could make use of logic apps to trigger an email. The list of files is still in the array variable which can be passed as request body which triggers the logic app for further processing - like including in the email body. Came across this step-by-step article that may help.

    You could gracefully exit the pipeline - if you want to fail the pipeline explicitly you could use one of the methods referenced in this thread

    136927-image.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators