Azure Data Factory -> Using Metadata activity + Filter

Question

I have a pipeline as seen in the screenshot below, the Filter activity below looks for CSV files + starting with a pattern as I specified + filters out the file if it contains the current date.

Condition below: @ANDO (and(and(equals(item().type,'File'),not(equals(item().name,concat('BBN_TX1_',formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'),'.csv')))),startswith(item().name,'BBN_TX1_')),endswith(item().name,'.csv'))

My requirement: I want to Filter out/Ignore any files which are of the current date or greater. I was able to achieve my requirement for the current date but for files greater than the current date I need help, please can anyone advise me on what can be done?

Answer 1

Hi @Vivek Komarla Bhaskar ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As I understand your ask, you want to filter out the filename based on the specified conditions : Filename should startwith 'BBN_TX1_' and endswith '.csv' and Filename should not contain dates greater than or equal to currentdate.

Assuming all the filenames are in same format i.e. 'BBN_TX1_yyyy-MM-dd.csv' , Kindly try out the below expression to achieve the requirement:

@and(and(and(and(equals(item().type,'File'),not(equals(item().name,concat('BBN_TX1_',formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'),'.csv')))),startswith(item().name,'BBN_TX1_')),endswith(item().name,'.csv')),not(greaterOrEquals(substring(item().name,8,10),formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'))))

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators