Remove-AzureRmDataFactoryPipeline is not deleting ADF pipeline present under GIT branch

Question

Remove-AzureRmDataFactoryPipeline is not deleting ADF pipeline present under GIT branch

RATH, ARUN 26

I have a requirement to delete 50 pipelines from my ADF. I tried powershell way using the below command :-

Remove-AzureRmDataFactoryV2Pipeline -Name "XYZ_DEF" -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName

I am getting "True" as response but I can still see the pipeline in my ADF under GIT branch.

Please let me know if powershell is the correct approach or there is any other smarter way to perform the same activity.

Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T07:11:20.17+00:00

Pipelines that you want to delete is in Master branch or some other manually created individual branch?
RATH, ARUN 26 Reputation points

2020-08-19T07:18:31.183+00:00

Pipelines are present at Dev branch which is different from Master Branch.
Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T07:29:42.447+00:00

PS commands ran successfully and returned true. Did it delete pipelines from master branch directly? Do cross check once
RATH, ARUN 26 Reputation points

2020-08-19T08:08:17.07+00:00

I checked it again. My Development branch under Azure Devops GIT is not getting deleted, however the pipelines are getting deleted from the DataFactory. PFA the screenshot for details :-
Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T08:52:27.507+00:00

As I mentioned in the answer, it will directly delete it from main factory (master branch). Currently there is no way you can pass the branch name and instruct PS command to delete from specific branch

Accepted answer

3 additional answers

Your answer

Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T07:11:20.17+00:00

Pipelines that you want to delete is in Master branch or some other manually created individual branch?
RATH, ARUN 26 Reputation points

2020-08-19T07:18:31.183+00:00

Pipelines are present at Dev branch which is different from Master Branch.
Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T07:29:42.447+00:00

PS commands ran successfully and returned true. Did it delete pipelines from master branch directly? Do cross check once
RATH, ARUN 26 Reputation points

2020-08-19T08:08:17.07+00:00

I checked it again. My Development branch under Azure Devops GIT is not getting deleted, however the pipelines are getting deleted from the DataFactory. PFA the screenshot for details :-
Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T08:52:27.507+00:00

As I mentioned in the answer, it will directly delete it from main factory (master branch). Currently there is no way you can pass the branch name and instruct PS command to delete from specific branch

Answer 1

MartinJaffer-MSFT 26,236

Hello @RATH, ARUN & @Vaibhav Chaudhari .
Sorry for the confusion. Please let me clarify.

Currently, GIT is affected only when using the visual authoring tool (adf.azure.com).

Powershell, SDK's, and API's push changes directly to the Data Factory service. This means they publish the change directly, bypassing any GIT integration.

Using Powershell to modify your Data Factory, while it is integrated with GIT, will result in changes to the 'Data Factory Mode' when using the visual authoring tool, but no changes applied to the GIT Mode.

Thank you,
Martin

RATH, ARUN 26 Reputation points

2020-08-20T05:08:31.877+00:00

Thank You @MartinJaffer-MSFT for a thorough explanation.

Can you let me know the best way to delete 50+ ADF pipeline linked to GIT (Developer Branch)... or I have to perform the activity Manually ?

Answer 2

HarithaMaddi-MSFT 10,146

Thank you @MartinJaffer-MSFT and @Vaibhav Chaudhari .

@RATH, ARUN - Please let me know if ADO approach (Azure DevOps) to manage pipelines that can be leveraged for CI/CD as well helps for your requirement. Once repository is cloned to our local, we can delete pipelines at a go and push them to Developer branch as shown in below GIF. I have used visual studio for cloning, this can be done using GIT commands as well.

Additional Info on CI/CD for ADF can be found in this link. We are also working with internal teams to understand alternate approaches and will share with you if this is not helpful.

RATH, ARUN 26 Reputation points

2020-08-24T13:23:36.967+00:00

Thank You for the GIF. It was really helpful.

Answer 3

Vaibhav Chaudhari 38,921 Volunteer Moderator

When GIT is enabled, PS command should directly update the objects in Data Factory mode.

Below is the note from document - ADF - source-control

Authoring directly with the Data Factory service is disabled in the Azure Data Factory UX when a Git repository is configured. Changes can be made directly to the service via PowerShell or an SDK.

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

RATH, ARUN 26 Reputation points

2020-08-19T09:22:17.3+00:00

GIT is enable in our case. And our collaboration branch is Developer(not the default one - Master) and publish branch is adf_publish.

As per your Answer, when we delete a pipeline from Power Shell command, it must delete it from Master(collaboration) branch.

As our collaboration branch is Developer, pipeline must be deleted from it but that's not happening.

As per the current scenario, Power Shell script is only deleting pipeline from DataFactory and not from Azure Devops GIT.

Please let me know the command where PS script can delete from Azure Devops GIT(collaboration branch) or if there is any other technique to delete multiple pipeline from collaboration branch.
Vaibhav Chaudhari 38,921 Reputation points Volunteer Moderator

2020-08-19T10:20:36.003+00:00

In my opinion, PS commands should be directly deleting pipelines from data factory mode.

I have done the pipelines deletion myself but it was manual way. Create a new branch, delete pipelines from UI, push changes from new branch to adf_publish and then master branch etc..

I would wait for MSFT support members to correct or let know if there is any way to delete pipelines from GIT branch
MartinJaffer-MSFT 26,236 Reputation points

2020-08-20T18:49:39.24+00:00

@Vaibhav Chaudhari

The note in the docs is poorly worded, at least as I understand. Here is how I read it:

Authoring directly with the Data Factory service is disabled in the Azure Data Factory UX when a Git repository is configured.

Once Git is enabled, you can look at the already-published resources using Data Factory Mode, however you are prohibited from altering them in this view. To make changes via the UX, you must use Git Mode, and only publish from the collaboration branch (often master branch).

Changes can be made directly to the service via PowerShell or an SDK.

The service refers to the Data Factory Service, those resources used when a pipeline is triggered, (not debug). Those resources resulting from a publish.

In my opinion the note is not very clear and I see how it easily leads to misunderstanding. I would like to change the document. To do that, I need your help. If you could open a doc issue, we can work together to ensure clarity.
MartinJaffer-MSFT 26,236 Reputation points

2020-08-26T21:55:18.82+00:00

Created pull request to clarify document.
https://github.com/MicrosoftDocs/azure-docs-pr/pull/127415

Answer 4

RATH, ARUN 26

I accomplished the task of deleting pipeline by :-

Creating a clone of the Dev branch in my local Visual studio.
By deleting all .json file (corresponding to the pipeline I wanted to delete) from newly cloned local branch.
Commit, Sync and Push the change to Dev GIT branch.

This is the easiest way I could find to delete 50+ pipelines in one shot.

Share via

Remove-AzureRmDataFactoryPipeline is not deleting ADF pipeline present under GIT branch

3 additional answers

Your answer