Fetch file names based on last modified date using synapse pipeline

Question

Hi Team,

We have some .txt files in ADLS Gen2. The requirement is to process the files in the order it arrived (Based on the last modified time). PFB the screen shot.

Could you please guide how to sort the files (using synapse pipeline activities) based on the lastmodifieddate ascending and pass the file name to subsequent pipeline / activity for processing.

Thanks.

Regards,
Nagesh CL

Answer 1

Hi @Nagesh CL ,

Thank you for posting query in Microsoft Q&A Platform.

One idea, which I am getting is, use Getmetadata activity to fetch all file names and then use ForEach activity to loop each file name. Inside Foreach activity, again use Getmetadata activity to get last modified date of file and then use script activity to insert that information into some SQL table.

Once above completes, all file names along with last modified dates information will be available in SQL table.

Now, you can execute a query on top of that SQL table to get rows in ascending order of last modified dates and then pass them to subsequent pipeline in same order.

Please check below useful videos or resources that help you to get idea about various activities and scenarios which may involve in above implementations.

• GetMetaData activity in Azure data factory
• ForEach activity in Azure data factory
• Script activity in Azure data factory
• Get File names from folder

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.