Get metadata filtering using a filename

Question

Greetings
I am trying to use get metadata activity to get only specific files with a date I want in my dataset from an FTP storage. File example is H_ODP_20201027_1232_00.CSV. I want to dynamically get files with certain date when I use the Tumbling window trigger so I can be able to load historical data. The metadata activity should only look at files with the specified date instead of going through all files in the folder as this causes a limitation error of 4MB. The ultimate goal is to copy all files filtered in the get metadata activity based on the filename date. I could have used copy by last modified but need to use the date in the filename as it represents the correct date the file was created. The image below shows how I hardcoded to get day 27 H files. I want to pass this date as a TWT parameter so it gets filtered. Thanks

Answer 1

Hi @Tinashe Chinyati ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Data factory supports the system variables as in this document which can be used in passing as parameters to the filename of dataset. Below code is an example to convert date into the format as desired.

@formatDateTime(utcnow(),'yyyyMMdd')  

Date functions can be used to modify the dates according to trigger date. For custom parameters that are not dependent on trigger time, solution needs to be designed to retrieve parameter value from any other source like blob/SQL using lookup activity in the pipeline.

Please let us know for further queries and we will be glad to assist.

--

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.