Azure Data Factory -> Using Metadata activity + Filter

Vivek Komarla Bhaskar 641 Reputation points

I have a pipeline as seen in the screenshot below, the Filter activity below looks for CSV files + starting with a pattern as I specified + filters out the file if it contains the current date.



Condition below: @ANDO (and(and(equals(item().type,'File'),not(equals(item().name,concat('BBN_TX1_',formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'),'.csv')))),startswith(item().name,'BBN_TX1_')),endswith(item().name,'.csv'))

My requirement: I want to Filter out/Ignore any files which are of the current date or greater. I was able to achieve my requirement for the current date but for files greater than the current date I need help, please can anyone advise me on what can be done?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,172 questions
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 23,271 Reputation points Microsoft Employee

    Hi @Vivek Komarla Bhaskar ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

    As I understand your ask, you want to filter out the filename based on the specified conditions : Filename should startwith 'BBN_TX1_' and endswith '.csv' and Filename should not contain dates greater than or equal to currentdate.

    Assuming all the filenames are in same format i.e. 'BBN_TX1_yyyy-MM-dd.csv' , Kindly try out the below expression to achieve the requirement:

    @and(and(and(and(equals(item().type,'File'),not(equals(item().name,concat('BBN_TX1_',formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'),'.csv')))),startswith(item().name,'BBN_TX1_')),endswith(item().name,'.csv')),not(greaterOrEquals(substring(item().name,8,10),formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'))))

    Hope this will help. Please let us know if any further queries.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

0 additional answers

Sort by: Most helpful