ADF : Read multiple files from source and write into single file in blob in append mode

Sanjeev Kumar Halladavar 30

In ADF , Requirement is to read multiple files and create single file in ADLS (Appended all read files).

Currently I see blob only supports overwrite not the append. Any auggestions to achive same in ADF

Nandan Hegde 36,151 Reputation points MVP Volunteer Moderator

2024-12-11T13:57:11.8+00:00

Do all the source files have same schema?

And when you say append files, do you mean append files from source into a single file or append source files with destination file
Sanjeev Kumar Halladavar 30 Reputation points

2024-12-11T14:10:11.5333333+00:00

Hi Nandan,

Yes, All files follow the same schema and naming convention file_name_<timestamp>.

Say, I have 10 files in adls folder, From these 10 files, i want to create one single file.

Accepted answer

Nandan Hegde 36,151 Reputation points MVP Volunteer Moderator

2024-12-11T14:18:45.25+00:00

please follow the below blog :

https://www.sqlservercentral.com/articles/merge-multiple-files-in-azure-data-factory

this would merge all files into single file
Please sign in to rate this answer.

1 person found this answer helpful.
Sanjeev Kumar Halladavar 30 Reputation points

2024-12-12T07:15:20.69+00:00

Thank you Nandan.

This works when we give all files/wildcard path/listoffiles from folder.

My requiremet is, Say, we have N number of files, from which we select couple of files and pass those files one by one in For loop and append to a file.

To Achive this, currently I am using GetMetadata activity, filter out required files and pass one file at a time to copy activity inside for loop and append to a file.

GetMetadata (ChildItems) --> ForEach (One file at a time) --> CopyActivity --> Consolidatedfile.txt

But it helped and let me tweak my approach a little bit.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

0 additional answers

Your answer