Copy files from SFTP to ADLS if files not present in ADLS with help of ADF

ashish ranjan 46 Reputation points
2022-08-16T18:01:40.69+00:00

Hi Team,

I need to copy files from SFTP(Source)to ADLS(Destination)if files not present in ADLS with help of ADF.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,662 questions
Accepted answer
  1. ShaikMaheer-MSFT 29,881 Reputation points Microsoft Employee
    2022-08-18T10:35:16.73+00:00

    Hi @ashish ranjan ,

    Thank you for posting query in Microsoft Q&A Platform.

    Below are the detailed steps of implementation, that helps to compare source and sink file names and filter out missing files and then loop them to copy.

    232335-image.png

    Here we are using two get meta data activities to get file names from source and sink locations. After that we are using filter activity to filter the file names which are missing in sink location. Writing expression correctly in Filter activity is important thing here. Once we filter the file names then we can use for each activity to loop them and inside for each activity use copy activity to copy file to destination.

    Below are the settings used in Filter activity to filter file names which are missing in sink location.
    232307-image.png

    Hope this helps. Please let us know if any further queries.

    ------------

    Please consider hitting Accept Answer button. Accepted answers help community as well.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Nandan Hegde 22,196 Reputation points
    2022-08-17T04:38:29.73+00:00

    Hey,
    You can use Get Meta data activity at the source and Get meta data activity in destination to get the chile items meaning the list of files present in source and destination.

    Than you can use Dataflow Join transformations to identify the missing file present in source and not in destination or in case if you want to avoid dataflow, you can use for each activity and IF activity :
    https://stackoverflow.com/questions/56435952/collection-to-exclude-array-items-in-azure-data-factory