ADF - Copy activity - getting list of files copied

Question

Is there a way in which I can get the list of files copied by the copy activity. I do not want to use the GetMetaData activity, as in the time different between executing the copy activity and the getmetadata activity, more files could have been added to the directory. Please suggest, thanks.

Basically in another activity after the copy, I have to log the files that have been copied.

Answer 1

Hi @Kothai Ramanathan ,

Thanks for posting the query.

I discussed with Product team and as @Vaibhav Chaudhari mentioned, they confirmed that there is no direct feature in Azure Data Factory to get the copied list. Following are the workarounds suggested. I understand the challenge in re-designing the code to achieve this requirement and I would recommend you to post an idea in feedback forum. The ideas posted here are closely monitored by ADF product team and they will work on implementing them.

• Using Get Metadata on the target folder to get the list of files copied but that can lead to payload size issue if there are large number of files
• Using Foreach and Get Metadata activity to track individual file name
• Using incremental load of files from source folder as that helps in retrieving the files picked based on the trigger time
• Using Azure functions to add blob trigger on target folder and to capture the list of files

Hope this helps!

Answer 2

Currently Copy Activity itself doesn't have any feature that gives list of files that got copied so there is no way right now to achieve this

Get metadata is only the workaround here

34132684-how-we-can-find-the-file-names-in-a-data-factory-c
get-files-list-after-azure-data-factory-copy-activity

---

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav