question

youssef125-3012 avatar image
0 Votes"
youssef125-3012 asked KranthiPakala-MSFT commented

ADF Copy activity inside foreach activity overwrite content in the sink

I'm new to ADF and I'm using a copy activity inside a foreach activity. the problem which i'm facing is every time the foreach activity executes, the copy activity overwrites the content in the sink(file stored in a container).

My question is, is there a way to make copy activity to add the content at the end of the file instead of overwriting data please ?

Thanks for you help

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @youssef125-3012,

Welcome to Microsoft Q&A forum and thanks for posting your ask.

Yes, Copy activity will overwrite the files if you provide the same file name in your sink. Copy activity doesn't support append data to existing file.

You don't need a forEach activity if your source files resides in same source container and your sink is a file based store. You can use wild card naming for your source dataset to pick the files and use just one copy activity that Merges all files when you chose copyBehavior as MergeFiles. This feature Merges all files from the source folder to one file. If the file name is specified, the merged file name is the specified name. Otherwise, it's an autogenerated file name.

145131-image.png

If you would like to write all data to a single file from multiple sources then you will have to write the initial files for each iteration to an intermediate folder by appending datetime to the filename and then once all the iterations are completed then have another subsequent copy activity outside of your ForEach activity and point it's source to the intermediate folder and in sink point to your desired file store and then use copyBehavior` as MergeFiles in copy activity settings.

Here is the public doc related to this feature : Copy activity - File system as sink - copyBehavior

Hope this info helps. Do let us know if you have further query.



  • Please don't forget to click on 130616-image.png and upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (81.0 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you @KranthiPakala-MSFT for your reply. Yes this is the only solution exists now is use the merge file options. by the way I'm usinfg foreach activity because it iterates an array and pass the value to the copy activity then the copy activity call an api at the source and store the date at the sink in container with a different name the I add a different copy activity to merge the fime but the problem with approach is, I'm facinf some performance issues it takes much time to merge the files


0 Votes 0 ·

Hi @youssef125-3012 , Glad to know above solution was helpful. As your follow-up query is related to performance and different from the original ask, could you please open a new thread so that it will have better visibility to community members.

Thank youj

0 Votes 0 ·