Azure Data Factory using data flow with delta format

asked 2021-11-23T22:13:25.747+00:00
Raoul Mahadeen 1 Reputation point

I'm currently trying to load data from Azure data lake v2 in delta format to a azure sql data warehouse. I have it loading but I need to figure out how to filter the source so I'm not grabbing all the data from the source (azure data lake v2) I want to be able to say something like " Give me all the data that has CreatedDate (column) between getdate()_1 and getdate()
I tried this
between($DateCreated, toDate("2021-11-16 00:00:00"), toDate("2021-11-16 01:00:00"))

or
between($DateCreated, addDays(currentTimestamp(),-1), currentTimestamp())

but get this error

DF-EXP-061 at Source 'source1'(Line 2/Col 27): Column operands are not allowed in literal expressions - RunId: e7794b09-634d-47d0-8a3a-777a9939d096

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,254 questions
{count} votes

1 answer

Sort by: Most helpful
  1. answered 2021-11-24T16:56:10.757+00:00
    ShaikMaheer-MSFT 28,096 Reputation points Microsoft Employee

    Hi @Raoul Mahadeen ,

    Thank you for posting query in Microsoft Q&A Platform.

    You can use Filter transformation and provide condition there. You can use condition with > && < as shown in below image.

    In below image "doj" is my column name
    152288-image.png

    Hope this will help. Please let us know if any further queries.

    ----------------------------------

    Please consider hitting Accept Answer button if this helps. Accepted answers helps community as well.