Share via

How to make the Chekpoint Key / SAP to stage subscriber process a static value?

Georgi Ivanov 0 Reputation points
2023-12-09T11:16:12.3066667+00:00

Context: I am currently building a pipeline that extracts data from a SAP CDC source, where I want to utilize the delta mechanism this connector enables. My setup is:

  • SAP CDC connector to source system (SAP CRM)- static subscriber name and ODP context.
  • A dedicated dataset (not yet parameterized) that connects to an ODP object on the SAP side. ODP context and ODP name are constants
  • A Data Flow utilizing the aforementioned dataset together with a sink to .parquet in an ADLS Blob Storage V2

Pipeline summary

Setup of the data flow:Data flow

In the data flow setup, I have entered a desired checkpoint key, which I want to be static. The reason I want it static is that, I want to connect to the same SAP ODP subscription every time, i run/debug the pipeline. As far as I understand it, it is the checkpoint key together with the subscriber name that maintains the relation to the SAP subscriptions.

Issue: Every time I make changes to the pipeline (not related to the data flow), this checkpoint is overwritten with the static value I have put in the property and a hash afterwards. Like so:User's image

The result of this is that I create a new subscription on the SAP side, every time I run this pipeline. Therefore, each load is becoming an Initial load.

As per Microsoft documentation- Checkpoint key, this key is a hash of pipeline name and data flow name. In my case no change occurs in either of the two.

Furthermore, I cannot permanently enable the "Override" option in the data flow setup. It resets every time I open/close the pipeline.

Question: How do I make this key static and a constant, so that I can maintain delta synchronization via the same subscription?

Attempts to resolve: In the dataset's .json definition, I have created the following property:

User's image

I know that this custom key is already related to an existing subscription on the SAP side.

This doesn't seem to give any positive result.

Expected outcome: I want to understand how to set the Checkpoint Key's value to a static value that does not change in runtime.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Bröthaler Thomas 5 Reputation points
    2025-07-01T13:51:58.1933333+00:00

    This is actually intentional since microsoft does not want you to accidentally overwrite your checkpoint key when debugging. Therefore they automatically append this suffix (I think based on your debug or browser session) to your customized checkpoint key. When you want to actually test or use the customized checkpoint key you need to push to the main branch and publish your changes into the live-mode.

    It took me a quite a while to find the documentation: https://docs.azure.cn/en-us/data-factory/sap-change-data-capture-advanced-topics#debugging

    Edit: For some reason as of today (2025-08-21) this behavior seem to have changed! Now, the customized checkpoint key is completely ignored for debug runs and therefore you cannot restart an initial load at will. In order to actually test CDC you need to run your data flow in the live-mode via a triggered run. But be careful, that you do not use your production checkpoint key (best to add the environment and an incremental number).

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments
  2. Amira Bedhiafi 42,126 Reputation points MVP Volunteer Moderator
    2023-12-09T14:46:04.45+00:00

    Based on what you provided , I think iInstead of using JSON definition to set a static value, you could define a global parameter at the pipeline level and use this to set the checkpoint key. This way, the key remains consistent across different executions as long as the global parameter value does not change.

    Also, If the global parameters do not work as expected, you might look into using pipeline templates. This feature allows you to define certain aspects of your pipeline that remain constant between executions.

    If the UI does not provide a way to set a static checkpoint key, you could consider using the Azure Data Factory REST API to programmatically set the checkpoint key to a desired value every time you run the pipeline. This way, you can ensure that the key does not change.

    Another alternative is to use a custom activity before your data flow that sets the checkpoint key to a specific value. This activity could be an Azure Function or a piece of custom code running in a Batch service that updates the necessary properties in the ADF metadata.

    I would recommend the ARM (Azure Resource Manager) templates, they will allow you to define the properties of your data factory in a JSON format, which can be version-controlled and deployed consistently.

    https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/overview

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.