How to get the latest Json file from any Cloud source to the ADLS whenever the ADF Pipeline runs

kks8589 96

Hi Team
Whenever we run a pipeline in ADF, how we are getting the latest file from a list of files from a source and copying it into the ADLS

Regards
Kks

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-10-07T18:17:20.783+00:00

Hello ,
Thanks for question and also for using this new forum .

Please check this link https://learn.microsoft.com/en-us/azure/data-factory/solution-template-copy-new-files-lastmodifieddate see it helps .

Thanks Himanshu
kks8589 96 Reputation points

2020-10-08T05:03:39.413+00:00

Hi Himanshu
Thanks for the link. I can see that in Copy new files only by LastModifiedDate template we have support for below sources as well as destinations

And also we should give the parameters last modified from and to manually everytime. My source is GCS and when ever i run the pipeline it should pick only the latest one

1 answer

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-10-08T23:04:04.617+00:00

Hello @kks8589 ,

Thanks for the reply . .
I am not sure as to why the template is not showing GCS , let me share the pipeline which i tested ( the logic is pretty much same ) .
Lets talk about the logic , we are going to run the pipeline using tumbling window trigger and it has two parameters start & EndTIme, and these are system varibles Starttime will be sent set as the time when the last time trigger ran EndTime as to be the current runtime .

So to your ask of do we need to pass start and End time manually , that should not be the case .

The below snapshot should help .

Lets configure the triiger now and publish the pipeline .

Thanks Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Please sign in to rate this answer.
kks8589 96 Reputation points

2020-10-14T06:58:24.51+00:00

HI Himanshu
If my GCS bucket is having 5 latest files uploaded after the last pipeline run and the tumbling window trigger run is of 5 days after then we will be getting all the 5 files into the datalake right?

Regards
kks

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-10-14T21:42:21.657+00:00

Hello @kks8589 ,

Since the TWT is configured to look into the bucket and pickup the file which are in the start and end time range , it will pickup all the files . How frequently are you expecting files to arrive/created on the GCS side ?

Thanks
Himanshu

kks8589 96 Reputation points

2020-10-19T05:11:15.64+00:00

Thats not the requirement, it should not get all the files. not sure on the frequency they are placing files in GCS however whenever they run the adf pipeline it may be once in fifteen days or a month or a week. when ever it run , it should fetch only the latest file that is there

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-10-28T18:07:12.86+00:00

Thanks @kks8589 ,
My apoloziges for the late reply .
Can you please through this implementation and it should work out . The implamentation should work for json files also .
https://learn.microsoft.com/en-us/answers/questions/136815/index.html

Thanks
Himanshu
Sign in to comment

Share via

How to get the latest Json file from any Cloud source to the ADLS whenever the ADF Pipeline runs

1 answer