Change data capture in Synapse

Mathias Opland 110 Reputation points


I want to use change data capture feature in Synapse Data Flow to avoid transferring the same data multiple times. However, I have to questions regarding this feature:

  • I can not see that it's available with Delta-files, only parquet. Is this something that's going to be available with Delta format?
  • When using git, the Checkpoint Key becomes a part of the commit if the pipeline has been running. I'm afraid that running the pipeline in my development environment will change the key, and upon deploying to production will create problems. Do you know if this is the case, or if the synapse environment will maintain the key even though code with a different key is deployed?


Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,825 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 45,902 Reputation points Microsoft Employee

    @Mathias Opland Welcome to Microsoft Q&A forum and thanks for reaching out here.

    1. Regarding CDC feature in mapping data flow: Yes, it is limited to Parquet and other data sources called out in this documentation: ASFAIK it is not in the roadmap at this point, I recommend submitting feedback in User voice forum here: Please do share the feedback link once it is posted so that others with similar idea can upvote and comment on your feedback to help increase the priority of the feature request.
    2. Regarding the checkpoint key question w.r.t GIT enabled workspace - As per my conversation with Product team, Checkpoint key is just a key, which is used to store the value in ADF backend, whereas checkpoint marker is stored as value against that key. Unless key is not changed by user/customer, things are supposed to work.

    Hope this info helps.

    Thank you

    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful