When deploying Azure data factory how to stop CDC pipeline using pre script

Question

When deploying Azure data factory how to stop CDC pipeline using pre script

Sanchila, Namindu 5

Hi,

I am currently facing an issue when deploying the Azure data factory using CDC.

User's image

To stop triggers I am using a prescript provided by Microsft document. How do I improve that pre-script to stop CDC pipeline automatically?

Here is big picture of the my pipeline User's image

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-08T22:40:22.6433333+00:00

@Sanchila, Namindu Thanks for using Microsoft Q&A forum and posting your query.

To better assist on this query, could you please help share the complete error message that you are receiving. Looks like that error message has specific details about the root cause of the problem. The image doesn't have complete details. Also please help share the prescript that you used to stop the triggers.

How do I improve that pre-script to stop CDC pipeline automatically? - Do you mean to stop the trigger that triggers CDC implementation pipeline or an already running pipeline?

Thank you

Sanchila, Namindu 5

@KranthiPakala-MSFT Here is the full error message:

2023-09-11T08:15:07.5052326Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
2023-09-11T08:15:07.5065666Z ##[error]Details:
2023-09-11T08:15:07.5066564Z ##[error]UpdateADFDataMapperStatusConflict: ADF CDC cdcdwb cannot be updated in its current status: 'Started'. Please stop the CDC first, publish your changes and then start it again.
2023-09-11T08:15:07.5067502Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
2023-09-11T08:15:07.5068487Z ##[error]Task failed while creating or updating the template deployment.

How do I improve that pre-script to stop CDC pipeline automatically? - Do you mean to stop the trigger that triggers CDC implementation pipeline or an already running pipeline?

No, there is a feature called Change Data Capture (Preview), when we are doing deployment to Test env, all deployments are failing.

User's image

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-13T22:54:21.29+00:00

@Sanchila, Namindu Thanks for the clarification. Seems the prescript is focused on triggers to stop the pipeline. Whereas CDC itself is a different feature.

I'm checking with respective product owners on the best practices to be followed in this scenario and will get back to you as soon as I hear back from the team.

Thank you
Sanchila, Namindu 5 Reputation points

2023-09-14T09:18:30.5633333+00:00

@KranthiPakala-MSFT ,

Thank you, how long it will take you to get back to us?
Sanchila, Namindu 5 Reputation points

2023-09-19T01:51:56.9966667+00:00

@KranthiPakala-MSFT ,

Thank you for your help. I will try this and let you know.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-25T17:41:33.9033333+00:00

Sounds good. Please keep this thread posted so that others will get benefit of it.

2 answers

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-08T22:40:22.6433333+00:00

@Sanchila, Namindu Thanks for using Microsoft Q&A forum and posting your query.

To better assist on this query, could you please help share the complete error message that you are receiving. Looks like that error message has specific details about the root cause of the problem. The image doesn't have complete details. Also please help share the prescript that you used to stop the triggers.

How do I improve that pre-script to stop CDC pipeline automatically? - Do you mean to stop the trigger that triggers CDC implementation pipeline or an already running pipeline?

Thank you
Sanchila, Namindu 5 Reputation points

2023-09-12T02:00:45.61+00:00

@KranthiPakala-MSFT Here is the full error message:

2023-09-11T08:15:07.5052326Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. 2023-09-11T08:15:07.5065666Z ##[error]Details: 2023-09-11T08:15:07.5066564Z ##[error]UpdateADFDataMapperStatusConflict: ADF CDC cdcdwb cannot be updated in its current status: 'Started'. Please stop the CDC first, publish your changes and then start it again. 2023-09-11T08:15:07.5067502Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting 2023-09-11T08:15:07.5068487Z ##[error]Task failed while creating or updating the template deployment.

How do I improve that pre-script to stop CDC pipeline automatically? - Do you mean to stop the trigger that triggers CDC implementation pipeline or an already running pipeline?

No, there is a feature called Change Data Capture (Preview), when we are doing deployment to Test env, all deployments are failing.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-13T22:54:21.29+00:00

@Sanchila, Namindu Thanks for the clarification. Seems the prescript is focused on triggers to stop the pipeline. Whereas CDC itself is a different feature.

I'm checking with respective product owners on the best practices to be followed in this scenario and will get back to you as soon as I hear back from the team.

Thank you
Sanchila, Namindu 5 Reputation points

2023-09-14T09:18:30.5633333+00:00

@KranthiPakala-MSFT ,

Thank you, how long it will take you to get back to us?
Sanchila, Namindu 5 Reputation points

2023-09-19T01:51:56.9966667+00:00

@KranthiPakala-MSFT ,

Thank you for your help. I will try this and let you know.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-25T17:41:33.9033333+00:00

Sounds good. Please keep this thread posted so that others will get benefit of it.

Answer 1

@Sanchila, Namindu Thank you for your patience.

Yes, it is possible to start/stop the running CDCs as part of pre- and post-automatically. For reference, I have the sample scripts (attached) which can get list of all running CDCs for a factory and stop them as part of prescript and then later start them again as part of postscript.

PS: You would need to pass the stopped CDCs names from prescript to postscript as task output environment variables.

Hope this info help. Please let me know how it goes.

Answer 2

Sudip Bala 20

Sanchila, Namindu. Did you resolved this ?

Sudip Bala 20 Reputation points

2024-05-09T12:06:12.42+00:00

I am facing the same issue KranthiPakala-MSFT

Share via

When deploying Azure data factory how to stop CDC pipeline using pre script

2 answers

Your answer