Processing incremental data changes

Important

Some or all of this functionality is available as part of a preview release. The content and the functionality are subject to change.

Incremental load uses the Change Data Feed (CDF) capability in Microsoft Fabric Lakehouse to capture only records that are inserted, updated, or deleted since the last refresh. By ingesting this delta rather than performing a full data reload, you dramatically reduce compute overhead and shorten processing windows. This pattern lowers operational costs and ensures downstream analytics and reporting always reflect the most current state of your Lakehouse without the resource effect of full reloads.

The ProcessCdfTable function

The ProcessCdfTable function, defined in the PREFIX_NDS_Config notebook, supports this incremental processing approach. It uses Delta Lake’s Change Data Feed (CDF) to identify new or modified rows in a source table and applies those changes to a target table.

This function fits into a Medallion architecture (for example, bronze, silver, and gold layers) and helps ensure data is updated efficiently through tracking changes, enriching records, and handling updates and inserts (upserts).

Key concepts

Concept Description
Change Data Feed (CDF) Enables reading only the changed rows (inserts/updates) from a Delta table since a specific commit version.
Commit Version Tracking Maintains a record of the last processed _commit_version using an internal Delta table (ChangeDataFeedState).
Surrogate Key Enrichment Optionally enriches records with dimension keys or other business logic through a user-defined enrich_func.
Idempotent Merge Executes a parameterized MERGE INTO SQL template to insert or update the target table with processed data.

Processing workflow

This section outlines the incremental processing approach.

Prerequisites

The source Delta table must have Change Data Feed enabled:

  • ALTER TABLE <table_name> SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
  • The ChangeDataFeedState tracking table must exist in the target lakehouse. The table is created automatically if not present.

Workflow steps

  1. Determine the last processed version The system checks the ChangeDataFeedState table for the most recent _commit_version processed for a given source/target table pair.

  2. Read changes from the Delta source Using Delta Lake's readChangeFeed option, the function reads only new or updated records since the last known version. Deletes and preimages are excluded.

  3. Apply the deduplication logic If multiple updates exist for the same primary key, only the latest change is selected per _commit_version.

  4. Optionally enrich the data If the table specifies enrich_func, the filtered data is passed through this function for dimension lookups or other transformation logic.

  5. Execute the merge logic The processed snapshot is materialized into a temporary view, and the defined merge_sql_template is executed to apply the changes to the target table.

  6. Update the processing state After successful processing, the function records the most recent commit version to the ChangeDataFeedState table for future executions.

See also

  • Last updated on