question

pankajchaturvedi-6530 avatar image
0 Votes"
pankajchaturvedi-6530 asked pankajchaturvedi-6530 commented

Check the Files are updated in the Datalake within the last day using ADF

Hi Team,

How can I check in Azure Datalake my files have been updated in the last day using ADF.?

Check: 'File has been updated within the last day .'If not , do not run and pass an error. "File Name" has not been updated within last day and I need to make sure that "File Name" is updated and need to run "ADF" manually.

Please help me out if do you have any idea.

Many thanks,
Pankaj

azure-data-factoryazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

NandanHegde-7720 avatar image
1 Vote"
NandanHegde-7720 answered

Hey,
You can use GetMeta Data activity to get the file update details and work out your need

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Nandan,

Thanks for your response.

Using GetMeta data activity only can retrieve last modified date but can not set the flag. As i want to throw the error message also when file doesn't update.

Could you please share screenshots how did you implemented if possible.Thanks!

0 Votes 0 ·
NandanHegde-7720 avatar image NandanHegde-7720 pankajchaturvedi-6530 ·

Hey,
You need to manage a config table or a config file to keep the tab of the last details.
You can then use it to compare the latest data from Get meta data and update the flag and either proceed as expected or use fail activity to fail the pipeline

1 Vote 1 ·

Hi Nandan,

Thanks for your response.

I didn't get you why i needed config table or config file.? Could you please let me know how did you implemented?
Or Share the screenshots so i can follow the same.Thanks!

0 Votes 0 ·

Hi Nandan,

Could you please let me know how did you update the flag value and compare the latest data with the Get metadata? Thanks!!

0 Votes 0 ·
ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered pankajchaturvedi-6530 commented

Hi @pankajchaturvedi-6530 ,

Thanks for posting query in Microsoft Q&A Platform.

You can consider using GetMetaData activity & If activity for achieving this.

GetMetaData activity can help to get LastModified date of file and then take that date in to If activity and validate and proceed pipeline execution accordingly.

This video will help to understand GetMetadata activity.
This video will help to understand If activity.

Hope this helps. Please let us know if any queries.


Please consider hitting Accept Answer button. Accepted answers help community as well.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @pankajchaturvedi-6530 ,

Just checking in to see if the above answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi ShaikMaheer,

Thanks for your response.

I am aware about Get Metadata and IF activity but not able to understand how can I set the flag as I want to throw the

error message when file doesn't update. Could you please share the screenshots how did you implemented .? Thanks!!

0 Votes 0 ·