How to do an incremental load WITHIN ADLS using data WITHIN a file LSN/change date-time (project como)

JHowe 1 Reputation point
2021-10-26T09:47:33.827+00:00

Hi all,

We have data landing in ADLS gen 2 from D 365 in CDM format (project como) we want to explore the possibility of creating an additional incremental layer between where the data is landing and our data tool but cannot find any decent examples/documentation to do this. The only documentation i can find is based on using copy data tool https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool, https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-partitioned-file-name-copy-data-tool. However this is based on either last modified date or file name. We want to incrementally load data using records WITHIN the change feed files i.e. LSN, change date-time. How is this possible? What is the best way to do this?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,527 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,119 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.