ADF - Copy activity - getting list of files copied

Kothai Ramanathan 906 Reputation points Microsoft Employee
2020-09-01T10:11:49.237+00:00

Is there a way in which I can get the list of files copied by the copy activity. I do not want to use the GetMetaData activity, as in the time different between executing the copy activity and the getmetadata activity, more files could have been added to the directory. Please suggest, thanks.

Basically in another activity after the copy, I have to log the files that have been copied.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,317 questions
No comments
1 vote

Accepted answer
  1. HarithaMaddi-MSFT 10,076 Reputation points
    2020-09-02T07:55:24.143+00:00

    Hi @Kothai Ramanathan ,

    Thanks for posting the query.

    I discussed with Product team and as @Vaibhav Chaudhari mentioned, they confirmed that there is no direct feature in Azure Data Factory to get the copied list. Following are the workarounds suggested. I understand the challenge in re-designing the code to achieve this requirement and I would recommend you to post an idea in feedback forum. The ideas posted here are closely monitored by ADF product team and they will work on implementing them.

    • Using Get Metadata on the target folder to get the list of files copied but that can lead to payload size issue if there are large number of files
    • Using Foreach and Get Metadata activity to track individual file name
    • Using incremental load of files from source folder as that helps in retrieving the files picked based on the trigger time
    • Using Azure functions to add blob trigger on target folder and to capture the list of files

    Hope this helps!

    No comments

1 additional answer

Sort by: Most helpful
  1. Vaibhav Chaudhari 37,776 Reputation points
    2020-09-01T12:55:43.467+00:00

    Currently Copy Activity itself doesn't have any feature that gives list of files that got copied so there is no way right now to achieve this

    Get metadata is only the workaround here

    34132684-how-we-can-find-the-file-names-in-a-data-factory-c
    get-files-list-after-azure-data-factory-copy-activity


    Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

    No comments