Validate files in ADLS using Logic app or Azure function for invoking ADF pipeline

Question

I get csv files in ADLS folder at different intervals and expected file list are available Azure SQL Table. how can I validate that all the files are arrived in adls using any of the below options
option1 - validate using logic app and invoke Azure Data factory.
option2 - validate using azure function and invoke Azure Data factory

Answer 1

Hi Karthikeyan Shivasankaran,
Thanks for question. I believe there is not direct or easy way to achieve it using ADF out of box features.
Below are few workarounds you may try:

1. Maybe you can write a custom code to check the file names in ADLS folder against your SQL table data to make sure that all files have arrived. This custom code can be executed using Custom activity in ADF. One the custom activity confirms all files are present then have a subsequent copy activity to copy all those files, If they don't have all the files you can skip the activity flow. You can use event triggers to trigger the pipeline when a new file arrives and pass the file names to your custom activity using GetMetada activity.
2. Another work around is to use Event trigger to trigger your Parent pipeline which will have a GetMetadata activity which will get all the files names in that folder and next have a Lookup activity to get list of expected files form your SQL table and then do a condition check using dynamic expression to validate if the expected file names from SQL table are present in the Item list from the GetMetadata activity output and if the matches then have a child pipeline that will have a copy activity and copies all the expected files. That way you can monitored two pipelines independently and it will be easy for troubleshooting in any odd scenarios. The advantage of having 2 pipelines (parent (validates if all the expected files have arrived in source) and child (executes copy operation on expected files)) is you can make sure to trigger the Core/Main ADF pipeline on when all the expected files have arrived or landed in your source.

I understand this is a tricky requirement, but if you want to avoid the custom code, then it would take multiple steps to implement this in ADF itself.

Hope this helps.