Azure Data Factory - Can you paramaterise CDC?

Mark Mann 20 Reputation points
2024-01-16T16:43:27.7333333+00:00

Looking at the public preview feature of Change Data Capture (CDC) in Azure Data Factory it would be really neat if you could pass parameters into this but I see no option to do this like you can for pipelines? Is this feature on the roadmap/coming soon?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,236 questions
0 comments No comments
{count} votes

Accepted answer
  1. phemanth 13,785 Reputation points Microsoft Vendor
    2024-01-17T14:02:35.21+00:00

    @Mark Mann
    Thanks for posting your question in the Microsoft Q&A forum.

    Azure Data Factory does not support parameterization in the Change Data Capture (CDC) feature directly. However, you might consider a workaround by creating a pipeline that wraps your CDC activity. This pipeline could accept parameters and use them to dynamically modify the linked service or dataset properties before invoking the CDC activity.

    Here’s a high-level example of how this could work:

    {
        "name": "PipelineWrapper",
        "properties": {
            "parameters": {
                "connectionString": {
                    "type": "string"
                }
            },
            "activities": [
                {
                    "name": "ModifyLinkedService",
                    "type": "AzureFunctionActivity",
                    "linkedServiceName": {
                        "referenceName": "AzureFunctionLinkedService",
                        "type": "LinkedServiceReference"
                    },
                    "typeProperties": {
                        "functionName": "ModifyLinkedService",
                        "method": "post",
                        "body": {
                            "linkedServiceName": "YourLinkedService",
                            "connectionString": "@pipeline().parameters.connectionString"
                        }
                    }
                },
                {
                    "name": "CDCActivity",
                    "type": "Copy",
                    "linkedServiceName": {
                        "referenceName": "YourLinkedService",
                        "type": "LinkedServiceReference"
                    },
                    "typeProperties": {
                        "source": {
                            "type": "CosmosDbSqlApiSource"
                        },
                        "sink": {
                            "type": "CosmosDbSqlApiSink"
                        }
                    }
                }
            ]
        }
    }
    

    In this example, an Azure Function is used to modify the linked service with the new connection string before the CDC activity is invoked. Please note that this is a simplified example, and the actual implementation might be more complex depending on your specific requirements and environment. Remember to check the official Azure Data Factory documentation for the most accurate and up-to-date information, https://learn.microsoft.com/en-us/azure/data-factory/concepts-change-data-capture https://learn.microsoft.com/en-us/azure/data-factory/concepts-change-data-capture-resource Hope this helps. Do let us know if you any further queries.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. Aleksei Zhukov 0 Reputation points
    2024-01-16T17:14:49.6333333+00:00

    What kind of parameter do you want to pass to CDC? The idea of CDC is to configure data source and wait until the changes (update or new items) will be added. The exact configuration (let's say folder aor key column) depends on Source linked service type


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.