Copy files from SFTP to ADLS if files not present in ADLS with help of ADF

Question

Hi Team,

I need to copy files from SFTP(Source)to ADLS(Destination)if files not present in ADLS with help of ADF.

Answer 1

Hi @ashish ranjan ,

Thank you for posting query in Microsoft Q&A Platform.

Below are the detailed steps of implementation, that helps to compare source and sink file names and filter out missing files and then loop them to copy.

Here we are using two get meta data activities to get file names from source and sink locations. After that we are using filter activity to filter the file names which are missing in sink location. Writing expression correctly in Filter activity is important thing here. Once we filter the file names then we can use for each activity to loop them and inside for each activity use copy activity to copy file to destination.

Below are the settings used in Filter activity to filter file names which are missing in sink location.

Hope this helps. Please let us know if any further queries.

------------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Answer 2

Hey,
You can use Get Meta data activity at the source and Get meta data activity in destination to get the chile items meaning the list of files present in source and destination.

Than you can use Dataflow Join transformations to identify the missing file present in source and not in destination or in case if you want to avoid dataflow, you can use for each activity and IF activity :
https://stackoverflow.com/questions/56435952/collection-to-exclude-array-items-in-azure-data-factory