Change data capture in Synapse

Question

Hi,

I want to use change data capture feature in Synapse Data Flow to avoid transferring the same data multiple times. However, I have to questions regarding this feature:

• I can not see that it's available with Delta-files, only parquet. Is this something that's going to be available with Delta format?
• When using git, the Checkpoint Key becomes a part of the commit if the pipeline has been running. I'm afraid that running the pipeline in my development environment will change the key, and upon deploying to production will create problems. Do you know if this is the case, or if the synapse environment will maintain the key even though code with a different key is deployed?

Regards
Mathias

Answer 1

@Mathias Opland Welcome to Microsoft Q&A forum and thanks for reaching out here.

1. Regarding CDC feature in mapping data flow: Yes, it is limited to Parquet and other data sources called out in this documentation: https://learn.microsoft.com/en-us/azure/data-factory/concepts-change-data-capture. ASFAIK it is not in the roadmap at this point, I recommend submitting feedback in User voice forum here: https://feedback.azure.com/d365community/forum/9b9ba8e4-0825-ec11-b6e6-000d3a4f07b8 Please do share the feedback link once it is posted so that others with similar idea can upvote and comment on your feedback to help increase the priority of the feature request.
2. Regarding the checkpoint key question w.r.t GIT enabled workspace - As per my conversation with Product team, Checkpoint key is just a key, which is used to store the value in ADF backend, whereas checkpoint marker is stored as value against that key. Unless key is not changed by user/customer, things are supposed to work.

Hope this info helps.

Thank you

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.