Azure Data Factory - clean partition folder before load

Question

Hello,

I'm creating a process that will load daily data into ADLS. The data will be partitioned by Date column value (date in format YYYY-MM-DD).

The root folder name is: Test

If for example I load June 13th and June 14th data, it'll automatically create two partition folders: "Date=2023-06-13" and "Date=2023-06-14" under the Test folder.
The question is how can I clean the particular date folder before the next load?
If in the next run I'll load the June 14th and June 15th data, the "Date=2023-06-14" to be cleaned before the load of the new June 14th file and the new folder "Date-06-15" to be created.

The "Clear the folder" option cannot be used as it'll delete all the subfolders (partitions) under the Test folder.

Answer 1

Hi BGO,

Thank you for posting query in Microsoft Q&A Platform.

Did you tried clear folder setting in transformation? If not try and see. If that not helps, then you need to consider having a seperate logic before processing data, to identify dates from data and check of partition paths for same and clean them. Once clean done, we can process our data to load.

Hope this helps. Please let me know how it goes. Thank you.