Row tracking in Databricks

Row tracking allows Azure Databricks to track row-level lineage in a table. Some incremental updates for materialized views require this feature.

All Apache Iceberg v3 tables include row tracking. For Delta Lake tables, you must explicitly enable row tracking.

Important

Row tracking is available in Databricks Runtime 14.1 and above.

Row tracking is a table feature and uses a higher table writer protocol than some clients. Table protocol versions can't be downgraded, and tables with row tracking enabled aren't writeable by clients that don't support all enabled writer protocol table features. See Delta Lake feature compatibility and protocols.

Enable row tracking on Delta tables

To enable row tracking on a Delta Lake table, set the table property delta.enableRowTracking = true during table creation:

CREATE TABLE table_name
TBLPROPERTIES (delta.enableRowTracking = true)
AS SELECT * FROM source_table;

To enable row tracking on an existing Delta Lake table, use the following example:

ALTER TABLE table_name SET TBLPROPERTIES (delta.enableRowTracking = true);

Important

Enabling row tracking on existing tables automatically assigns row IDs and row commit versions to all existing rows in the table. This process can result in the creation of multiple new versions of the table and could take a significant amount of time.

Cloning a table creates a separate history, so the row IDs and row commit versions on cloned tables don't match those in the original table.

What is the schema of the row tracking metadata fields?

Row tracking adds two hidden metadata fields to the table. You can explicitly add these fields to your query to return the values.

Column name	Type	Values	Explanation
_metadata.row_id	Long	The unique identifier of the row.	A row keeps the same ID whenever it is modified using a MERGE or UPDATE statement.
_metadata.row_commit_version	Long	The Delta log or table version at which the row was last inserted or updated.	A row is assigned a new version whenever it is modified using a MERGE or UPDATE statement.

Some operations store these metadata fields using the transaction log. Running OPTIMIZE or REORG operations on a table with row tracking enabled rewrites data files to store these fields.

Disable row tracking on Delta tables

To disable row tracking on a Delta Lake table, set the table property to false.

ALTER TABLE table_name SET TBLPROPERTIES (delta.enableRowTracking = false);

Important

Disabling row tracking doesn't remove the corresponding table feature and doesn't downgrade the table protocol version. It also doesn't remove the metadata fields from the target table.

With row tracking disabled, the row IDs generated are no longer reliable for tracking unique rows.

Limitations

The row IDs and row commit versions metadata fields can't be accessed while reading the change data feed. See Use Delta Lake change data feed on Azure Databricks.