azure data factory get metadata not filtering by last modified date

SAMB 21 Reputation points
2021-06-25T13:20:19.667+00:00

I'm trying to find if a file exists in blob storage using the get metadata function, but only if has been modified in the last 5 minutes.. I am using the following settings:

109339-1.png

This sends the below json:

`{  
    "dataset": {  
        "referenceName": "LandingExcel",  
        "type": "DatasetReference",  
        "parameters": {  
            "FilePath": "2021/06/25",  
            "FileName": "2019.xlsx"  
        }  
    },  
    "fieldList": [  
        "exists",  
        "lastModified"  
    ],  
    "storeSettings": {  
        "type": "AzureBlobStorageReadSettings",  
        "recursive": true,  
        "modifiedDatetimeStart": "2021-06-25T12:57:45.9744278Z",  
        "modifiedDatetimeEnd": "2021-06-25T13:02:45.9744278Z",  
        "enablePartitionDiscovery": false  
    }  
}`  

But the response is still showing a file existing even though it is outside of the 5 minute window:

{ "exists": true, "lastModified": "2021-06-25T09:50:45Z", "effectiveIntegrationRuntime": "VNet-IR", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 80 }, "billingReference": { "activityType": "PipelineActivity", "billableDuration": [ { "meterType": "ManagedVNetIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }

Am I misunderstanding the functionality or doing something wrong? Is there a different way to filter the metadata response by a time period?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,526 questions
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-06-28T07:02:12.15+00:00

    Hi @SAMB ,

    Welcome to Microsoft Q&A Platform. Thank you for posting query here.

    Exists filed in GetMetaData will tell you if your dataset is exists or not, Irrespective of Filter by last modified values.

    You need to use GetMetadata activity with a dataset pointing to your folder(2021/06/25) and use filter by Last modified values for last 5 minutes and use Child items field. This will help you child items which are modified in last 5 minutes.

    In below example, I am trying to take files which are modified in last 5 minutes from "data" folder and then ForEach activity to loop those files and IF activity to check my file(Employees.csv) is there.

    Step1: GetMetaData Activity to take files which are modified in last 5 minutes from data folder

    109775-getmetadata.gif
    Step2: ForEach Activity to iterate your ChildItems

    109753-foreachactivity.gif
    Step3: IF Activity to check "Employees.csv" file exists or not

    109776-ifactivity.gif
    Step4: IF fail exists then under True use your activities accordingly.

    Hope this will help. Thank you

    ------------------------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications
    3 people found this answer helpful.

0 additional answers

Sort by: Most helpful