Merge Multiple Files in Order using Azure Data Factory

Question

Hi,

I have a scenario to merge all the files from a folder while copying to another location using 'copy activity'. In the Sink activity, I am selecting the copy behavior as 'Merge Files' and this is merging all the files. But, when I look at the data inside the merged file, all the records are not in order. Instead, I am trying to the read the files in order and merge them in order without changing the data/rows order of each file. Please suggest a better way to achieve this.

Thanks,
Jeevi

Answer 1

Hello @Jeevi Danda ,

Thanks for the question and using MS Q&A platform.

As of today, the ADF copy mergeFiles behavior doesn't have control over the order of files to be merged and also the order of rows to be merged. It considers randomly.

If you have feedback regarding this, I recommend submitting a feature request for the same in ADF IDEAS forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

All the feedback shared in this forum are actively monitored and reviewed by respective product owners. Please do share the feedback link once it is posted so that I shall up-vote it and also will pass the feedback to respective product owners.

Hope this info helps.