Data parameterization in the data flow within Azure data factory | Source options - "Fiter by last modified" start time (UTC)

Question

Hi,

We are trying to parameterize the dates ( Source options - "Filter by last modified" start time (UTC)) in the data factory- data flow - but its not allowing us to fetch the expected result.

The date parameterization logic is not getting picked up and is getting trespassed with NO EROORS.

Input date format: 2021-04-12 04:53:00.0000000

Logic applied - toTimestamp($start_date, 'yyyy-mm-dd\'T\'HH:mm:ss')

Answer 1

Hi @Singh, Kumar Deepesh ,
If your Input date format: 2021-04-12 04:53:00.0000000 then you need to use 'yyyy-MM-dd HH:mm:ss.SSS' in your toTimeStamp function, so you could try something like this - toTimestamp('2021-04-12 04:53:00.000', 'yyyy-MM-dd HH:mm:ss.SSS') in your code to convert it to required format.

In order to use Time format like toTimestamp($start_date, 'yyyy-mm-dd\'T\'HH:mm:ss') your input date format should be 2021-04-12T04:53:00.
Please refer to the highlighted text -

Please refer to the documentation. I have also checked this in my environment it works as expected. (See the below gif)

Please let me know if you have any other questions.

---

Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

Answer 2

Were you able to solve this issue. I am trying to apply Filter By Last Modified in Data Flow where source is Azure Data Lake .

It seems the filtering logic is getting passed and all files are being retrieved.