Replace/overwrite using ADF on ADLS

Rama Jaddu 1 Reputation point
2021-09-06T15:12:00.387+00:00

We have ADF pipeline COPY activity to copy the data from Hive to ADLS.
We need replace/overwrite option on ADF copy for file/directory copy to ADLS gen2 .
currently it replaces only if we are inserting/writing with the same filename. we need to replace the old data file/folder with the new data file/folder irrespective of the filename.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,893 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,046 Reputation points
    2021-09-07T20:51:48.767+00:00

    Hello @Rama Jaddu and welcome to Microsoft Q&A.

    If I understand correctly, you want to replace sink folder, and files, even if it has different name.

    Suppose sink has contents:

    container  
    container/folderX/  
    container/folderX/fileX1  
    container/folderX/fileX2  
    container/folderZ/  
    container/folderZ/fileZ1  
    container/folderZ/fileZ2  
    

    and you want to write

    container/folderNew/  
    container/folderNew/Newfile1  
    container/folderNew/Newfile2  
    

    How would I know which branch to replace? Should folderX or folderZ be replaced? This is unclear.

    You can use the Delete activity to remove a folder and all its files.

    0 comments No comments