Share via

Data parameterization in the data flow within Azure data factory | Source options - "Fiter by last modified" start time (UTC)

Singh, Kumar Deepesh 6 Reputation points
2021-04-15T13:22:34.09+00:00

Hi,

We are trying to parameterize the dates ( Source options - "Filter by last modified" start time (UTC)) in the data factory- data flow - but its not allowing us to fetch the expected result.

The date parameterization logic is not getting picked up and is getting trespassed with NO EROORS.88244-capture-df1.png

Input date format: 2021-04-12 04:53:00.000000088293-capturedf2.png

Logic applied - toTimestamp($start_date, 'yyyy-mm-dd\'T\'HH:mm:ss')

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Jogalekar, Mandar 56 Reputation points
    2021-06-07T15:22:58.713+00:00

    Were you able to solve this issue. I am trying to apply Filter By Last Modified in Data Flow where source is Azure Data Lake .

    It seems the filtering logic is getting passed and all files are being retrieved.

    Was this answer helpful?

    0 comments
  2. Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator
    2021-04-29T01:46:35.497+00:00

    Hi @Singh, Kumar Deepesh ,
    If your Input date format: 2021-04-12 04:53:00.0000000 then you need to use 'yyyy-MM-dd HH:mm:ss.SSS' in your toTimeStamp function, so you could try something like this - toTimestamp('2021-04-12 04:53:00.000', 'yyyy-MM-dd HH:mm:ss.SSS') in your code to convert it to required format.

    In order to use Time format like toTimestamp($start_date, 'yyyy-mm-dd\'T\'HH:mm:ss') your input date format should be 2021-04-12T04:53:00.
    Please refer to the highlighted text -
    92364-image.png

    Please refer to the documentation. I have also checked this in my environment it works as expected. (See the below gif)
    92423-dataflow-datetimeformat.gif

    Please let me know if you have any other questions.

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.