How to validate source files in Azure data factory

Question

I am new to the Azure Data Factory. I have a requirement to validate the source file from a particular directory and check whether that particular file name exists in that path then perform the copy activity, if the file is a missing file in that directory then it should fail and send an email stating the input file not present in the source.

Steps followed: I have created a pipeline and added a "Get data activity" and selected the arguments as "Child List", now I am getting the list of files that are available in that particular directory.

The next steps would be is to check/validate whether the list of mandatory files for that particular Data source and matches the "Get met data activity" output and then perform the Copy activity.

Can you please guide me on how to validate it.

Note: The source file names will be appended with today's date at the end so need to trim it and compare the file names.

Hope my requirement is clear.

Thanks a lot for your help

Answer 1

Hi @Deepak G ,

Welcome to Microsoft Q&A Platform.

Below is one approach I have used to implement the above requirement partially. This can be enhanced to cover entire scenario.

• Took one config table - file check- that has static list of files and with status column as flag that can be reset before every load in pipeline. I took for testing initially flag as 0 for 3 files
• Fileload is the final table that loads data from all files in this case, this can be parameterized as per requirement to different tables/files
• Flag column can be used after looping of all the files to understand which files are missing and Logic App can be configured to trigger mail(Have not shown in below gif but this link explains it well)
• Trim and string functions can be used in "IF Condition" activity in below GIF that can trim the additional characters in the file before comparison

Hope this helps! Do let us know for further queries.