How to do an incremental load WITHIN ADLS using data WITHIN a file LSN/change date-time (project como)
Hi all,
We have data landing in ADLS gen 2 from D 365 in CDM format (project como) we want to explore the possibility of creating an additional incremental layer between where the data is landing and our data tool but cannot find any decent examples/documentation to do this. The only documentation i can find is based on using copy data tool https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool, https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-partitioned-file-name-copy-data-tool. However this is based on either last modified date or file name. We want to incrementally load data using records WITHIN the change feed files i.e. LSN, change date-time. How is this possible? What is the best way to do this?