I was able to work out the solution and apologies for the delay. I am now using 2 copy activities with dataset as binary. So since our files keep on accumulating in the source folder I was allowed to copy and delete the source files while copying them to another file system storage creating a date partition. It reads the datepart in the filename Q_ODP_20201218_2334_00.CSV. So if we want to load historic data we specify using the TWT. The sink path can be adjusted accordingly depending on the spec in this case its the same. The following was my resolution and seems to work. Thanks
how to copy files from file system to adlsgen2 in ADF
I am new to ADF. I have an on-premise storage that receives small files every min and I would like to copy these files based on the timestamp in the filename, then sink the respective file to the folder in adlsgen2 e.g. file format is H_ODG_20201206_213412_00.CSV and should be sinked in Year=2020, Month=12 and Day=06 etc, since there will be a lot of files with diff dates I want to be able to create a tumbling window trigger that filters what to copy based on that file name (timestamp). Thanks for your help
Sign in to comment
1 additional answer
Sort by: Most helpful
Hello @Tinashe Chinyati ,
Thanks for the ask and using the forum .
There are two system variables exposed by the TWT to capture the start and endtime . Please do read about that here .
And in the If activity with the below expression and I think thats all you need .
Let me know how it goes .
Hi @Tinashe Chinyati ,
With the 4MB limit , I think you have the only option is to use the lastModified date for the last 10/15/20 hrs and see if that pulls data less then 4 MB . I know that in your case the date part is the name of the file ( which you want to use ) , but then we do not have a filter defined on the Filename , so we will have to use the last modified filter in some way .
One other way ( may be it works for your )
If we can keep a track of the files which was copied the last time and use the last modified date of that file as an water mark to copy the new files . We can acomplish this by use a lookup .
We have not heard back from you on this and was just following up .
Incase if you have resolution , request you to share the same here , so that other community members can benefit from that .