ADF Copy activity inside foreach activity overwrite content in the sink

youssef125 266 Reputation points
2021-10-28T19:30:18.177+00:00

I'm new to ADF and I'm using a copy activity inside a foreach activity. the problem which i'm facing is every time the foreach activity executes, the copy activity overwrites the content in the sink(file stored in a container).

My question is, is there a way to make copy activity to add the content at the end of the file instead of overwriting data please ?

Thanks for you help

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,339 questions
0 comments No comments
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,412 Reputation points Microsoft Employee
    2021-10-29T20:44:52.677+00:00

    Hi @youssef125 ,

    Welcome to Microsoft Q&A forum and thanks for posting your ask.

    Yes, Copy activity will overwrite the files if you provide the same file name in your sink. Copy activity doesn't support append data to existing file.

    You don't need a forEach activity if your source files resides in same source container and your sink is a file based store. You can use wild card naming for your source dataset to pick the files and use just one copy activity that Merges all files when you chose copyBehavior as MergeFiles. This feature Merges all files from the source folder to one file. If the file name is specified, the merged file name is the specified name. Otherwise, it's an autogenerated file name.

    145131-image.png

    If you would like to write all data to a single file from multiple sources then you will have to write the initial files for each iteration to an intermediate folder by appending datetime to the filename and then once all the iterations are completed then have another subsequent copy activity outside of your ForEach activity and point it's source to the intermediate folder and in sink point to your desired file store and then use copyBehaviorasMergeFiles` in copy activity settings.

    Here is the public doc related to this feature : Copy activity - File system as sink - copyBehavior

    Hope this info helps. Do let us know if you have further query.

    ----------

    • Please don't forget to click on 130616-image.png and upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful