azure data factory get metadata not filtering by last modified date

Question

I'm trying to find if a file exists in blob storage using the get metadata function, but only if has been modified in the last 5 minutes.. I am using the following settings:

This sends the below json:

`{  
    "dataset": {  
        "referenceName": "LandingExcel",  
        "type": "DatasetReference",  
        "parameters": {  
            "FilePath": "2021/06/25",  
            "FileName": "2019.xlsx"  
        }  
    },  
    "fieldList": [  
        "exists",  
        "lastModified"  
    ],  
    "storeSettings": {  
        "type": "AzureBlobStorageReadSettings",  
        "recursive": true,  
        "modifiedDatetimeStart": "2021-06-25T12:57:45.9744278Z",  
        "modifiedDatetimeEnd": "2021-06-25T13:02:45.9744278Z",  
        "enablePartitionDiscovery": false  
    }  
}`  

But the response is still showing a file existing even though it is outside of the 5 minute window:

{ "exists": true, "lastModified": "2021-06-25T09:50:45Z", "effectiveIntegrationRuntime": "VNet-IR", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 80 }, "billingReference": { "activityType": "PipelineActivity", "billableDuration": [ { "meterType": "ManagedVNetIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }

Am I misunderstanding the functionality or doing something wrong? Is there a different way to filter the metadata response by a time period?

Answer 1

Hi @SAMB ,

Welcome to Microsoft Q&A Platform. Thank you for posting query here.

Exists filed in GetMetaData will tell you if your dataset is exists or not, Irrespective of Filter by last modified values.

You need to use GetMetadata activity with a dataset pointing to your folder(2021/06/25) and use filter by Last modified values for last 5 minutes and use Child items field. This will help you child items which are modified in last 5 minutes.

In below example, I am trying to take files which are modified in last 5 minutes from "data" folder and then ForEach activity to loop those files and IF activity to check my file(Employees.csv) is there.

Step1: GetMetaData Activity to take files which are modified in last 5 minutes from data folder

Step2: ForEach Activity to iterate your ChildItems

Step3: IF Activity to check "Employees.csv" file exists or not

Step4: IF fail exists then under True use your activities accordingly.

Hope this will help. Thank you

------------------------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications.