Copy Latest files from Azure blob storage

pankaj chaturvedi 86 Reputation points
2023-01-17T15:53:45.7633333+00:00

Hi Team,

I have multiples CSV files in my Azure blob storage container and I want to extract only the files which are updated/uploaded in the last 15 minutes in my azure blob storage and load into Azure datalake storage. Could you please help me out if someone has implemented.

I am looking forward your response. Many thanks.!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,537 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Manu Philip 16,971 Reputation points MVP
    2023-01-17T16:19:16.0533333+00:00

    Have you checked the answer provided in the below question?

    Copy last modified blob


    --please don't forget to upvote and Accept as answer if the reply is helpful--


  2. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2023-01-18T20:10:57.2833333+00:00

    Hi

    If your requirement is to just copy the files that were uploaded or modified in the last 15 minutes when your pipeline is being triggered, then the simplest solution would be to utilize *Filter by last modified* feature in copy activity Source tab.

    Please use below setting and it should help achieve your requirement. Please note I used *utcnow* in below example and used addMinutes function to look back for last 15 minutes. If you want to change the timezone or format, then you can update the dynamic expression as per your specific requirement. For more Date time functions please refer to this document: ADF Date functions for Dynamic expression.

    User's image

    Hope this helps. Do let us know if you have further questions.


    Please don’t forget to *Accept Answer* and *Up-Vote* wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments