rename of pipeline failed part-way, now I have both source and destination pipeline and cant remove them

Question

rename of pipeline failed part-way, now I have both source and destination pipeline and cant remove them

Robert Vaughan 0

Hi we have a pipeline A that was being renamed to pipeline B, and the rename has failed. now we have pipeline A with an EntityState of "Deleting", and Pipeline B with an EntityState of "Creating". they are linked by Source and Target values and an OperationId. e.g.

       "EntityState": "Deleting",
        "RenameOperationDetails": {
            "Source": "Pipeline A",
            "Target": "Pipeline B",
            "OperationId": "7aed9c51-9377-4a25-954e-682587898b41"

We can't rename. remove or update these pipelines, in any way, the message being "The Operation failed because the entity 'Pipeline A' is being renamed to 'Pipeline B'. A retry of rename from exact source to target may work"

It doesn't work, now it is blocking our deployment via GitHub as they were created directly in our Live environment and not via Git.

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-09T08:20:23.2633333+00:00
@robert vaughan

Thanks for using MS Q&A platform and posting your query.

I understand that you’re facing issues with renaming pipelines in Azure Synapse Analytics. This seems to be a common issue when pipelines are renamed directly in the live environment, especially when the workspace is Git configured

Here are a few steps you can try to resolve this issue:

Revert to the Original Name: Change the pipeline back to the original name and republish

Use a New Local Branch: Create a new local branch, rename the pipeline there, and then raise a PR to merge the changes into the collaboration branch

Clone the Pipeline: If directly changing the pipeline name is not working, clone the pipeline and provide the desired name for the new pipeline. Then, delete the older pipeline and merge the changes.

Hope this helps. Do let us know if you any further queries.
Robert Vaughan 0 Reputation points

2024-05-09T10:09:36.64+00:00

Hi @phemanth Thanks for your suggestions, however, our Prod Synapse instance isn't Git configured, only the dev/test one, so we are unable to create a branch. (we use CI/CD to push changes to our Prod synapse instance)

we are unable to rename either of the pipelines (A & B) , I keep getting the 'rename is in progress' message, and the publish fails.

I can clone the existing pipeline, but am still unable to remove it, getting the same error mentioned above.

We have tried using the CLI -both versions (az Synapse pipeline delete & remove-AzSynapsePipeline) but neither work. we've also tried updating the code both in the JSON editor in studio, and via the CLI.

Does anyone know of a 'backdoor' method to remove these objects, since the front-end tools aren't working?
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-10T09:17:35.15+00:00
@Robert Vaughan I understand that you’re facing challenges with renaming and removing pipelines in your Azure Synapse Analytics environment. Let’s explore some potential solutions to address this issue:

Cancel Running Pipelines:

Sometimes, a pipeline might be stuck in an in-progress state due to various reasons.

In the Synapse Studio, navigate to the Monitor tab.

Identify the pipeline run that is causing the issue (e.g., the one with the “rename is in progress” message).

Look for an option to cancel the run (usually represented by a “Cancel” button or similar UI element).

Confirm the cancellation of the run.

Check for Recovery Runs:

Occasionally, recovery runs can cause pipelines to hang.

Identify any recovery runs that might be interfering with your pipelines.

Cancel any such recovery runs to free up your pipeline for reruns.

Use PowerShell or Azure CLI:

Since the front-end tools aren’t working, consider using PowerShell or Azure CLI to manage your pipelines.

You mentioned trying the CLI commands az Synapse pipeline delete and remove-AzSynapsePipeline. Ensure that you have the necessary permissions and are using the correct syntax.

If these commands still don’t work, try using the following PowerShell script to remove the pipelines:

# Replace with your specific details $resourceGroupName = "YourResourceGroup" $workspaceName = "YourSynapseWorkspace" $pipelineNameA = "Pipeline A" $pipelineNameB = "Pipeline B" # Remove Pipeline A Remove-AzSynapsePipeline -ResourceGroupName $resourceGroupName -WorkspaceName $workspaceName -PipelineName $pipelineNameA # Remove Pipeline B Remove-AzSynapsePipeline -ResourceGroupName $resourceGroupName -WorkspaceName $workspaceName -PipelineName $pipelineNameB

Execute the script in your PowerShell environment, replacing the placeholders with your actual resource group, workspace, and pipeline namesUse

If none of the above methods work,please do let us know
Robert Vaughan 0 Reputation points

2024-05-10T14:30:57.8+00:00

@phemanth

Still no joy with those suggestions. thanks for trying though.

Rob.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-13T10:59:27.6966667+00:00
@Robert Vaughan

Azure Resource Explorer:

Azure Resource Explorer (ARE) is a powerful tool for directly interacting with Azure resources.

Open ARE and navigate to your Synapse workspace.

Locate the pipelines (A and B) and try deleting them from there.

Use the following URL format to access ARE:
https://resources.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}

Replace {subscriptionId}, {resourceGroupName}, and {workspaceName} with your actual values.

Azure REST API:

If ARE doesn’t work, consider using the Azure Synapse REST API.

You can programmatically delete pipelines using the API.

Refer to the official documentation for details on how to construct the request.

If none of the above methods work, please do let us know we will further help to resolve this issue.
Robert Vaughan 0 Reputation points

2024-05-14T08:23:26.24+00:00

@phemanth I couldn't find the pipeline via the ARE, even when using the URL you provided (also used Resource Explorer in the azure portal)

We also tried removing them via the REST API, but no luck there either.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-15T11:07:36.53+00:00
@Robert Vaughan

finally check these steps below

Azure CLI with Additional Flags:

While the az Synapse pipeline delete command didn't work previously, consider trying it again with the --force flag. This flag might bypass some checks and force the deletion, but use it with caution as it's a more aggressive approach.

Command:

az Synapse pipeline delete --resource-group $resourceGroupName --workspace-name $workspaceName --name $pipelineName --force

Important: Replace placeholders with your actual values and ensure you have the necessary permissions to use the --force flag.

Check for Locks:

In rare cases, resource locks might prevent deletion. Use Azure Portal or Azure CLI to check for any locks on the pipelines or the Synapse workspace. Remove any irrelevant locks to allow deletion.

if none of this work let us know we will check with our internal team
Darren Price 5 Reputation points

2024-07-30T22:33:16.45+00:00

Hi @Robert Vaughan did you ever resolved this? Hitting the exact same issue.
rob vaughan 0 Reputation points

2024-07-31T08:25:27.41+00:00

@darren price no, never resolved it, it's still causing problems for my CICD Pipeline!

Darren Price 5

Dam, I was hoping you would tell me the operation at least timedout after x number of days if you didn't find a fix.

I've tried Azure Cli, Az PowerShell, REST nothing appears to be able to force a way out of the of the situation.

Anyone else got around this issue? I think we need a way to terminate the operations that are stuck. 1 is stuck in a state of creating and 1 stuck in a state of deleting

"lastPublishTime": "2024-07-23T08:32:17Z",
"EntityState": "Creating", 
"RenameOperationDetails": { "Source": "Pipeline1", "Target": "Pipeline2", 
"OperationId": "088abbeb-0762-4bee-ac9b-1b1691c10b16" }

"lastPublishTime": "2024-07-23T08:32:17Z", 
"EntityState": "Deleting",
"RenameOperationDetails": { "Source": "Pipeline1", "Target": "Pipeline2", 
"OperationId": "088abbeb-0762-4bee-ac9b-1b1691c10b16" }

"status": "Failed", 
"error": { "code": "DeleteDataFactoryResourceOrchestrationError", 
"message": "The operation failed because the entity 'Pipeline1' is being renamed to 'Pipeline2'. A retry of rename from exact source to target may work." }

Johan 1 Reputation point

2024-09-26T09:54:31.3933333+00:00

Did either of you have any luck, @Darren Price or @rob vaughan ?

I've got a similar issue, but with a notebook, that's causing my CICD to fail.
Robert Vaughan 0 Reputation points

2024-09-26T10:00:14.65+00:00

no, still have this problem, I've turned off deletes in my cicd pipeline so I can still push changes through.
Darren Price 5 Reputation points

2024-09-26T10:27:44.4966667+00:00

Microsoft resolved via support ticket for us. No explanation how they resolved, I assume they are able force the stuck jobs to terminate.
Johan 1 Reputation point

2024-09-26T10:28:47.3666667+00:00

Thanks - I've sent a support ticket through to our CSP for resolution. My environment is completely bricked. Can't do any publishes at all.
Robert Cox 0 Reputation points

2024-10-12T14:54:20.5666667+00:00

We also had this issue occur on Thursday and we've been unable to resolve it and also had to resort to support ticket.

We have managed a semi fix by disconnecting the GIT connection and reconnecting and leaving the erroneous pipeline alone, which now lets us publish. Our solution is to leave that alone while MS support fix it, and recreate a new pipeline from scratch for our actual change.
Johan 1 Reputation point

2024-10-12T15:00:07.68+00:00

Microsoft resolved the issue for us, but the temporary workaround was quite simple, once we knew what it needed to be.

If you have an object which is stuck in this renaming, you just need to have zero changes affecting the object, and the environment will publish. This also allows CI/CD through DevOps, for example, as long as your CI/CD doesn't attempt to affect the faulty pipeline/notebook/object either through change or delete.

It was the delete that caught us out. There is a setting in the deployment task in Azure DevOps that you can disable, which stops the deployment task from deleting artifacts not included in the build in the target environment.

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-09T08:20:23.2633333+00:00

@robert vaughan

Thanks for using MS Q&A platform and posting your query.

I understand that you’re facing issues with renaming pipelines in Azure Synapse Analytics. This seems to be a common issue when pipelines are renamed directly in the live environment, especially when the workspace is Git configured

Here are a few steps you can try to resolve this issue:

Revert to the Original Name: Change the pipeline back to the original name and republish

Use a New Local Branch: Create a new local branch, rename the pipeline there, and then raise a PR to merge the changes into the collaboration branch

Clone the Pipeline: If directly changing the pipeline name is not working, clone the pipeline and provide the desired name for the new pipeline. Then, delete the older pipeline and merge the changes.

Hope this helps. Do let us know if you any further queries.
Robert Vaughan 0 Reputation points

2024-05-09T10:09:36.64+00:00

Hi @phemanth Thanks for your suggestions, however, our Prod Synapse instance isn't Git configured, only the dev/test one, so we are unable to create a branch. (we use CI/CD to push changes to our Prod synapse instance)

we are unable to rename either of the pipelines (A & B) , I keep getting the 'rename is in progress' message, and the publish fails.

I can clone the existing pipeline, but am still unable to remove it, getting the same error mentioned above.

We have tried using the CLI -both versions (az Synapse pipeline delete & remove-AzSynapsePipeline) but neither work. we've also tried updating the code both in the JSON editor in studio, and via the CLI.

Does anyone know of a 'backdoor' method to remove these objects, since the front-end tools aren't working?
Robert Vaughan 0 Reputation points

2024-05-10T14:30:57.8+00:00

@phemanth

Still no joy with those suggestions. thanks for trying though.

Rob.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-13T10:59:27.6966667+00:00

@Robert Vaughan

Azure Resource Explorer:

Azure Resource Explorer (ARE) is a powerful tool for directly interacting with Azure resources.

Open ARE and navigate to your Synapse workspace.

Locate the pipelines (A and B) and try deleting them from there.

Use the following URL format to access ARE:
https://resources.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}

Replace {subscriptionId}, {resourceGroupName}, and {workspaceName} with your actual values.

Azure REST API:

If ARE doesn’t work, consider using the Azure Synapse REST API.

You can programmatically delete pipelines using the API.

Refer to the official documentation for details on how to construct the request.

If none of the above methods work, please do let us know we will further help to resolve this issue.
Robert Vaughan 0 Reputation points

2024-05-14T08:23:26.24+00:00

@phemanth I couldn't find the pipeline via the ARE, even when using the URL you provided (also used Resource Explorer in the azure portal)

We also tried removing them via the REST API, but no luck there either.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-15T11:07:36.53+00:00

@Robert Vaughan

finally check these steps below

Azure CLI with Additional Flags:

While the az Synapse pipeline delete command didn't work previously, consider trying it again with the --force flag. This flag might bypass some checks and force the deletion, but use it with caution as it's a more aggressive approach.

Command:

az Synapse pipeline delete --resource-group $resourceGroupName --workspace-name $workspaceName --name $pipelineName --force

Important: Replace placeholders with your actual values and ensure you have the necessary permissions to use the --force flag.

Check for Locks:

In rare cases, resource locks might prevent deletion. Use Azure Portal or Azure CLI to check for any locks on the pipelines or the Synapse workspace. Remove any irrelevant locks to allow deletion.

if none of this work let us know we will check with our internal team
Darren Price 5 Reputation points

2024-07-30T22:33:16.45+00:00

Hi @Robert Vaughan did you ever resolved this? Hitting the exact same issue.
rob vaughan 0 Reputation points

2024-07-31T08:25:27.41+00:00

@darren price no, never resolved it, it's still causing problems for my CICD Pipeline!
Darren Price 5 Reputation points

2024-07-31T09:25:30.8566667+00:00

Dam, I was hoping you would tell me the operation at least timedout after x number of days if you didn't find a fix.

I've tried Azure Cli, Az PowerShell, REST nothing appears to be able to force a way out of the of the situation.

Anyone else got around this issue? I think we need a way to terminate the operations that are stuck. 1 is stuck in a state of creating and 1 stuck in a state of deleting

"lastPublishTime": "2024-07-23T08:32:17Z", "EntityState": "Creating", "RenameOperationDetails": { "Source": "Pipeline1", "Target": "Pipeline2", "OperationId": "088abbeb-0762-4bee-ac9b-1b1691c10b16" }

"lastPublishTime": "2024-07-23T08:32:17Z", "EntityState": "Deleting", "RenameOperationDetails": { "Source": "Pipeline1", "Target": "Pipeline2", "OperationId": "088abbeb-0762-4bee-ac9b-1b1691c10b16" }

"status": "Failed", "error": { "code": "DeleteDataFactoryResourceOrchestrationError", "message": "The operation failed because the entity 'Pipeline1' is being renamed to 'Pipeline2'. A retry of rename from exact source to target may work." }
Johan 1 Reputation point

2024-09-26T09:54:31.3933333+00:00

Did either of you have any luck, @Darren Price or @rob vaughan ?

I've got a similar issue, but with a notebook, that's causing my CICD to fail.
Robert Vaughan 0 Reputation points

2024-09-26T10:00:14.65+00:00

no, still have this problem, I've turned off deletes in my cicd pipeline so I can still push changes through.
Darren Price 5 Reputation points

2024-09-26T10:27:44.4966667+00:00

Microsoft resolved via support ticket for us. No explanation how they resolved, I assume they are able force the stuck jobs to terminate.
Johan 1 Reputation point

2024-09-26T10:28:47.3666667+00:00

Thanks - I've sent a support ticket through to our CSP for resolution. My environment is completely bricked. Can't do any publishes at all.
Robert Cox 0 Reputation points

2024-10-12T14:54:20.5666667+00:00

We also had this issue occur on Thursday and we've been unable to resolve it and also had to resort to support ticket.

We have managed a semi fix by disconnecting the GIT connection and reconnecting and leaving the erroneous pipeline alone, which now lets us publish. Our solution is to leave that alone while MS support fix it, and recreate a new pipeline from scratch for our actual change.
Johan 1 Reputation point

2024-10-12T15:00:07.68+00:00

Microsoft resolved the issue for us, but the temporary workaround was quite simple, once we knew what it needed to be.

If you have an object which is stuck in this renaming, you just need to have zero changes affecting the object, and the environment will publish. This also allows CI/CD through DevOps, for example, as long as your CI/CD doesn't attempt to affect the faulty pipeline/notebook/object either through change or delete.

It was the delete that caught us out. There is a setting in the deployment task in Azure DevOps that you can disable, which stops the deployment task from deleting artifacts not included in the build in the target environment.

Share via

rename of pipeline failed part-way, now I have both source and destination pipeline and cant remove them

Your answer