sink activity in dataflow

Question

arkiboys 9,706

hello,

using the sink activity in dataflow of data factory, I create _year, _month, _day partitions and add data in there for each day...

Question

How is it possible to delete everything in the daily partition before loading daily data into the daily partitions?

thanks

Subashri Vasudevan 11,306 Reputation points Volunteer Moderator

2023-10-09T13:28:15.32+00:00

arkiboys does it not replace the daily partition file when you do reload?
arkiboys 9,706 Reputation points

2023-10-09T14:26:56.46+00:00

no it does not,this is what I have:
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2023-10-10T05:49:07.5266667+00:00

@arkiboys Thanks for using Microsoft Q&A forum and posting your query.

Do you know upfront for which day partition the data has to be deleted? If you know upfront, then you can use the Delete activity in Azure Data Factory pipeline. In the Delete activity, set the Source path property to the daily partition path, including the year, month, and day values. Then set the Source path property to the daily partition path, including the year, month, and day values. Then add Data Flow activity to the pipeline and select the Data Flow that writes data to the daily partition. This way you run the pipeline to delete the data in the daily partition before writing new data to it.

Let me know if your requirement doesn't suffice with above implementation.

Thank you
arkiboys 9,706 Reputation points

2023-10-10T06:53:41.1+00:00

hi,

1- I am using delta parquet files with partitions as described above. With that can I still use the copy activity to delete the data in the know partition for year,month,day?

2- in the copy activity what do I put for sink?

thanks

Your answer

Subashri Vasudevan 11,306 Reputation points Volunteer Moderator

2023-10-09T13:28:15.32+00:00

arkiboys does it not replace the daily partition file when you do reload?
arkiboys 9,706 Reputation points

2023-10-09T14:26:56.46+00:00

no it does not,this is what I have:
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2023-10-10T05:49:07.5266667+00:00

@arkiboys Thanks for using Microsoft Q&A forum and posting your query.

Do you know upfront for which day partition the data has to be deleted? If you know upfront, then you can use the Delete activity in Azure Data Factory pipeline. In the Delete activity, set the Source path property to the daily partition path, including the year, month, and day values. Then set the Source path property to the daily partition path, including the year, month, and day values. Then add Data Flow activity to the pipeline and select the Data Flow that writes data to the daily partition. This way you run the pipeline to delete the data in the daily partition before writing new data to it.

Let me know if your requirement doesn't suffice with above implementation.

Thank you
arkiboys 9,706 Reputation points

2023-10-10T06:53:41.1+00:00

hi,

1- I am using delta parquet files with partitions as described above. With that can I still use the copy activity to delete the data in the know partition for year,month,day?

2- in the copy activity what do I put for sink?

thanks