Share via

dynamically replace the dates for every run in folder path

Vinodh247-1375 43,181 Reputation points Volunteer Moderator
2022-11-27T10:19:48.913+00:00

How to copy all the files from nested folders where the source folder name has date details in the middle.

I want the pipeline to dynamically fill the date item*(bolded in the path provided below) in the folder path every run and copy all the files from its nested folders. How can this be achieved?

For example, I receive files like below, I want to create a pipeline that would dynamically select and replace the date part only and execute for the files that has been received in sFTP location.

D:\dekshin\Temp\UnzippedFolder_20220906\firstfolder\USA\DB\DELIMITER
D:\dekshin\Temp\UnzippedFolder_20220913\firstfolder\USA\DB\DELIMITER
D:\dekshin\Temp\UnzippedFolder_20220920\firstfolder\USA\DB\DELIMITER
D:\dekshin\Temp\UnzippedFolder_20220927\firstfolder\USA\DB\DELIMITER

264450-image.png

**Updated the question for better understanding

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.


1 answer

Sort by: Most helpful
  1. Nitesh Chavan 81 Reputation points
    2022-11-28T05:27:59.2+00:00

    Maybe you can try something like this, I have tried this to add the current time (YYYYMMDD) in folder path as well as file name. Instead of replacing you can create your expected folder path.

    @markus.bohland@hotmail.de (pipeline().parameters.Par_LoadDetails.DstFileName),'_',formatDateTime(utcnow(),'yyyyMMdd'),'.parquet')
    @markus.bohland@hotmail.de (pipeline().parameters.Par_LoadDetails.DstFolderPath,'/',formatDateTime(pipeline().TriggerTime,'yyyyMMdd'))

    Also you can refer the below link for your reference,
    https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-partitioned-file-name-copy-data-tool

    Hope this helps.

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.