Merge Multiple Files in Order using Azure Data Factory

Jeevi Danda 71 Reputation points
2022-11-04T13:22:19.927+00:00

Hi,

I have a scenario to merge all the files from a folder while copying to another location using 'copy activity'. In the Sink activity, I am selecting the copy behavior as 'Merge Files' and this is merging all the files. But, when I look at the data inside the merged file, all the records are not in order. Instead, I am trying to the read the files in order and merge them in order without changing the data/rows order of each file. Please suggest a better way to achieve this.

Thanks,
Jeevi

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,933 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,432 Reputation points Microsoft Employee
    2022-11-08T01:53:00.313+00:00

    Hello @Jeevi Danda ,

    Thanks for the question and using MS Q&A platform.

    As of today, the ADF copy mergeFiles behavior doesn't have control over the order of files to be merged and also the order of rows to be merged. It considers randomly.

    If you have feedback regarding this, I recommend submitting a feature request for the same in ADF IDEAS forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

    All the feedback shared in this forum are actively monitored and reviewed by respective product owners. Please do share the feedback link once it is posted so that I shall up-vote it and also will pass the feedback to respective product owners.

    Hope this info helps.

    1 person found this answer helpful.