sink parquet

arkiboys 9,556 Reputation points
2022-08-31T05:29:57.717+00:00

hello,
I am using a copy activity.
source --> dataset - a dummy .csv file with one column and one value in column.
The source has additional columns which I added manually. This is the setting at the bottom of the source tab
sink --> dataset which is in .parquet

everytime I run the copy activity, the sink .parquet file gets replaced with the new .parquet and so the previous data disappears.
Question.
How is it possible to get multiple lines in my sink folder. perhaps how is it possible to have a separate parquet file for each run in sink?
Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,491 questions
0 comments No comments
{count} votes

Accepted answer
  1. Suba Balaji 11,186 Reputation points
    2022-08-31T05:54:34.27+00:00

    Hi @arkiboys ,

    Appending to the same file is not possible with copy activity.

    Alternatively, as you asked, you can create a new file everytime. For that create a parameter for filename in sink dataset and pass an expression like below from the copy activity sink.

    @markus.bohland@hotmail.de ('filename_',formatDateTime(utcNow(),'ddMMyyyyhhmmss'),'.csv')

    This will create a new file everytime your pipeline is run.

    Hope this helps. Let us know for any question.


0 additional answers

Sort by: Most helpful