Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,342 questions
Change Data Capture from Data Lake source using Common Data Model?
Kim Beck
20
Reputation points
I have some D365 sample data exported to Data Lake using Common Data Model.
In the Azure Data Factory Data Flow, I have configured a source to read from the manifest file under Changefeed folder and selected in this case the "HcmWorker" entity. The Dataset is an Inline Dataset.
There is an option for "Change Data Capture". How does this work in ADF using Data Lake files in Common Data model? How and what is tracked? e.g. is there a specific column the data flow uses to track what has been loaded/changed??
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee2023-01-30T04:08:05.6933333+00:00 Hi Kim Beck,
Welcome to Microsoft Q&A Forum and thank you for posting your query.
When using the change capture option for data flow sources, ADF will maintain and manage the checkpoint for you automatically. When you enable native change data capture or auto incremental extraction options in ADF mapping data flow, ADF helps you to manage the checkpoint to make sure each activity run will automatically only read the source data that has changed since the last time the pipeline run. The default checkpoint key is a hash of the data flow name and the pipeline name. If you are using a dynamic pattern for your source tables or folders, you may wish to override this hash and set your own checkpoint key value here. If you change your pipeline name or activity name, the checkpoint will be reset, which leads you to start from beginning or get changes from now in the next run. If you do want to change the pipeline name or activity name but still keep the checkpoint to get changed data from the last run automatically, please use your own Checkpoint key in data flow activity to achieve that. The naming rule of your own checkpoint key is same as linked services, datasets, pipelines and data flows. The naming rule of your own checkpoint key is same as linked services, datasets, pipelines and data flows.
Hope this info helps.
-
Kim Beck 20 Reputation points
2023-02-03T23:08:16.31+00:00 Thanks for your detailed reply.
I have managed to get my test running with incremental load, ticking off CDC. It seems to work as expected. I just need to understand what is going on behind the scene. :-)
- Where does ADF store the checkpoint key information? e.g. Can I look it up or query it? This would be useful in future for debugging and documentation cases.
-
Kim Beck 20 Reputation points
2023-02-03T23:08:49.12+00:00 - I assume that when I tick "CDC" in ADF for the Inline database, it is looking for the field "LastProcessedChange_DateTime" in the CDM table/entity definition?
ex:
"name": "LastProcessedChange_DateTime", "dataFormat": "DateTime", "description": "The DateTime timestamp for the last CDC change on this record."
Sign in to comment