ADF CICD Procedure

Question

I have implemented a cicd procedure for my adf. I have a dev and a prod one. In the CD procedure, I have three tasks: Stop triggers, ARM deployment and Start triggers. When I rename a pipeline in my dev adf or deleting a pipeline in dev adf it doesn't delete or rename it after the cicd procedure. It is only creating the new ones and when I make a rename, it keeps the old name and also creates a new one. When I also delete something in dev it doesn't delete it in prod also with the cicd. My cicd procedure uses this command for stopping the triggers and also the one for starting the triggers(I don't use the prepostdeploymentscript):

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName

$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.name -Force }

I also use linked templates for deployment.

Do I have to add the prepostdeploymentscript in order to have the appropriate result for renaming and deleting pipelines etc? If yes is it safe to use it? What is the result I should expect?

Also why are there two options for triggers?:

a ) $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName

$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.name -Force }

b) -armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $true -deleteDeployment $false

Answer 1

To handle renaming and deletion of pipelines effectively, you can utilize pre-deployment and post-deployment scripts. These scripts can manage the state of your ADF pipelines before and after the ARM template deployment. This approach ensures that any deletions or renames in the dev environment are mirrored in the prod environment.

The pre-deployment script can stop all triggers, delete all existing pipelines, and ensure a clean state before deploying the updated ARM template. This will prevent old pipelines from persisting after renames or deletions.

Here's an example of a pre-deployment script:

# Stop all triggers

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName

$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.name -Force }

# Delete all existing pipelines

$pipelinesADF = Get-AzDataFactoryV2Pipeline -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName

$pipelinesADF | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.name -Force }

# You can add more deletion commands for datasets, linked services..., if needed

After the pre-deployment script, you can deploy your ARM template. Here’s how you can structure the deployment:

New-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -TemplateFile "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -Mode Incremental -Name <DeploymentName>

Optionally, you can use a post-deployment script to start the triggers again after the deployment:

# Start all triggers

$triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.name -Force }

In your CI/CD pipeline, you can include these scripts as part of the deployment tasks. Here’s a generalized structure:

1. Pre-Deployment Task:
   • Stop triggers.
   • Delete existing pipelines (and other components if necessary).
2. ARM Template Deployment Task:
   • Deploy the ARM template with the new or updated pipelines.
3. Post-Deployment Task (Optional):
   • Start triggers.