Enabling ChangeDataFeed with ADF CDC

Question

Enabling ChangeDataFeed with ADF CDC

Xavier Geerinck 45

Hi All,

I am currently utilizing Azure Data Factory its Change Data Capture feature, which is sinking data to my Azure Blob Storage as a delta file.

How can I by default enable the enableChangeDataFeed parameter? That way I can create a realtime lakehouse.

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2025-05-12T10:12:56.0566667+00:00
Hi @Xavier Geerinck
To enable enableChangeDataFeed by default when using Azure Data Factory (ADF) with Delta format, you’ll typically need to ensure that the Delta table itself has this property turned on. ADF currently doesn’t expose a direct setting to enable Change Data Feed (CDF) during sink operations. This usually needs to be configured when the Delta table is created - commonly via Spark (e.g., in Databricks or Synapse) using:

ALTER TABLE [table_name] SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

If you're using ADF alone to write data, and not leveraging a Spark engine for Delta table creation, this property won’t be auto-enabled just by writing to Delta format via ADF.

Just to better understand your setup:

Are the Delta tables being created through another service (like Databricks or Synapse), or is ADF handling both creation and ingestion?

Also, is your end goal to stream or query changes in near real-time (e.g., using Structured Streaming or other tools downstream)?

Clarifying these points will help suggest the most practical way to integrate CDF into your lakehouse architecture.

Let us know and we will be glad to assist further!
Xavier Geerinck 45 Reputation points

2025-05-12T11:47:35.99+00:00
Thanks for the quick response! I am currently relying on ADF to create the table (initial load is still under investigation). An initial load is still possible (although the schema might be unknown and need to be evolved), though.

The main issue I am having is that when I try this manually through the below I get that Databricks always puts the reader and writer version on respectively 3c and 7 and Data Factory will then crash when starting the CDC writer. Trying to change it doesn't seem to be allowed.

What would be the recommended way towards getting a working CDC implementation on the medallion on Databricks? I am trying to incrementally load my data from SQL Server (Azure DB / Azure MI) -> Azure Blob Storage (using ADF CDC) <- Databricks (Managing Bronze -> Silver -> Gold through DLT and ReadStreams to incrementally do this)

# Create a new table spark.sql(f""" CREATE TABLE IF NOT EXISTS {TABLE_NAME} USING DELTA LOCATION '{TABLE_PATH_DELTA}' """) # Method 1 spark.conf.set("spark.databricks.delta.properties.defaults.minReaderVersion", "2") spark.conf.set("spark.databricks.delta.properties.defaults.minWriterVersion", "5") # Method 2 spark.sql(f"ALTER TABLE {TABLE_NAME} SET TBLPROPERTIES ('delta.enableChangeDataFeed' = true, 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5')") # Method 3 delta = DeltaTable.forPath(spark, TABLE_PATH_DELTA) delta.upgradeTableProtocol(2, 5)
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2025-05-13T11:08:07.07+00:00
Hi @Xavier Geerinck

Thanks a lot for the detailed explanation - that really helps clarify what you're trying to build.

From what you’ve described, it sounds like the main challenge is around enabling Change Data Feed (CDF) for Delta tables when using ADF’s CDC to write data. The tricky part here is that ADF doesn’t currently support setting the enableChangeDataFeed property by itself, and it also doesn’t handle the Delta protocol versions the same way Databricks does.

Here’s what usually works well in this setup:

Use Databricks to Create the Table First

Before ADF writes anything, it’s best to create the Delta table in Databricks, and explicitly enable Change Data Feed and set the right protocol version (reader v2 / writer v5). ADF expects these versions, and anything higher (like writer version 7) causes it to fail - which is likely what you’re seeing.

Here’s an example of how you can do it in Databricks:

spark.sql(f""" CREATE TABLE IF NOT EXISTS {TABLE_NAME} USING DELTA LOCATION '{TABLE_PATH_DELTA}' TBLPROPERTIES ( delta.enableChangeDataFeed = true, delta.minReaderVersion = 2, delta.minWriterVersion = 5 ) """)

Once the table is created like this, ADF should be able to write without crashing, and Databricks can pick up changes using CDF for your Bronze → Silver → Gold layers.

About Protocol Downgrades

You mentioned trying to change the protocol versions manually - unfortunately, once a Delta table has been written with a higher version (like v7), you can’t downgrade it. So, if the table was originally created by Databricks with the defaults, you’d need to:

Export the data

Recreate the table with the right version/properties

Load the data back in

Long-Term Suggestion:

Let ADF handle the initial data load (landing raw CDC files in Bronze) but always use Databricks to set up the Delta tables first, especially when you need features like CDF. That way you have full control over things like schema evolution, protocol versions, and table properties.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Xavier Geerinck 45 Reputation points

2025-05-13T12:51:04.9233333+00:00
Thanks for your response! I've looked into it again and got it working this time!!

The main issue is that Databricks will automatically default to the least support version. Now as deletionVectors are a more advanced feature, the query you shared doesn't work.

I will write a blog article later on how I got this working end-to-end, but as for now, changing it to the below worked:

DROP TABLE IF EXISTS customers; CREATE TABLE customers USING DELTA LOCATION 'abfss://masked/dbo.Customers' TBLPROPERTIES ( -- Disable deletion vectors as this doesn't work on 2.5 delta.enableDeletionVectors = false, delta.enableChangeDataFeed = true, delta.minReaderVersion = 2, delta.minWriterVersion = 5 )

Accepted answer

0 additional answers

Your answer

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2025-05-12T10:12:56.0566667+00:00

Hi @Xavier Geerinck
To enable enableChangeDataFeed by default when using Azure Data Factory (ADF) with Delta format, you’ll typically need to ensure that the Delta table itself has this property turned on. ADF currently doesn’t expose a direct setting to enable Change Data Feed (CDF) during sink operations. This usually needs to be configured when the Delta table is created - commonly via Spark (e.g., in Databricks or Synapse) using:

ALTER TABLE [table_name] SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

If you're using ADF alone to write data, and not leveraging a Spark engine for Delta table creation, this property won’t be auto-enabled just by writing to Delta format via ADF.

Just to better understand your setup:

Are the Delta tables being created through another service (like Databricks or Synapse), or is ADF handling both creation and ingestion?

Also, is your end goal to stream or query changes in near real-time (e.g., using Structured Streaming or other tools downstream)?

Clarifying these points will help suggest the most practical way to integrate CDF into your lakehouse architecture.

Let us know and we will be glad to assist further!
Xavier Geerinck 45 Reputation points

2025-05-12T11:47:35.99+00:00

Thanks for the quick response! I am currently relying on ADF to create the table (initial load is still under investigation). An initial load is still possible (although the schema might be unknown and need to be evolved), though.

The main issue I am having is that when I try this manually through the below I get that Databricks always puts the reader and writer version on respectively 3c and 7 and Data Factory will then crash when starting the CDC writer. Trying to change it doesn't seem to be allowed.

What would be the recommended way towards getting a working CDC implementation on the medallion on Databricks? I am trying to incrementally load my data from SQL Server (Azure DB / Azure MI) -> Azure Blob Storage (using ADF CDC) <- Databricks (Managing Bronze -> Silver -> Gold through DLT and ReadStreams to incrementally do this)

# Create a new table spark.sql(f""" CREATE TABLE IF NOT EXISTS {TABLE_NAME} USING DELTA LOCATION '{TABLE_PATH_DELTA}' """) # Method 1 spark.conf.set("spark.databricks.delta.properties.defaults.minReaderVersion", "2") spark.conf.set("spark.databricks.delta.properties.defaults.minWriterVersion", "5") # Method 2 spark.sql(f"ALTER TABLE {TABLE_NAME} SET TBLPROPERTIES ('delta.enableChangeDataFeed' = true, 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5')") # Method 3 delta = DeltaTable.forPath(spark, TABLE_PATH_DELTA) delta.upgradeTableProtocol(2, 5)
Xavier Geerinck 45 Reputation points

2025-05-13T12:51:04.9233333+00:00

Thanks for your response! I've looked into it again and got it working this time!!

The main issue is that Databricks will automatically default to the least support version. Now as deletionVectors are a more advanced feature, the query you shared doesn't work.

I will write a blog article later on how I got this working end-to-end, but as for now, changing it to the below worked:

DROP TABLE IF EXISTS customers; CREATE TABLE customers USING DELTA LOCATION 'abfss://masked/dbo.Customers' TBLPROPERTIES ( -- Disable deletion vectors as this doesn't work on 2.5 delta.enableDeletionVectors = false, delta.enableChangeDataFeed = true, delta.minReaderVersion = 2, delta.minWriterVersion = 5 )

Answer 1

Hi @Xavier Geerinck
I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Solution: The main issue is that Databricks will automatically default to the least support version. Now as deletionVectors are a more advanced feature, the query you shared doesn't work.

I will write a blog article later on how I got this working end-to-end, but as for now, changing it to the below worked:

DROP TABLE IF EXISTS customers;
CREATE TABLE customers
USING DELTA
LOCATION 'abfss://masked/dbo.Customers'
TBLPROPERTIES (
    -- Disable deletion vectors as this doesn't work on 2.5
    delta.enableDeletionVectors = false,
    delta.enableChangeDataFeed = true,
    delta.minReaderVersion = 2,
    delta.minWriterVersion = 5
)

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members

Share via

Enabling ChangeDataFeed with ADF CDC

0 additional answers

Your answer