Exporting Selected Azure Data Factory Pipelines

Question

Hi,

I have a question, I have Data Factory which has almost 850 pipelines.

Out of 850 pipelines. 250 of them are created using as Back up or created for versioning of pipelines.

now i want to take backup of those pipelines and want to store their ARM template in backup folder and delete them for ADF
one solution I have in mind is that i create a pipeline and put all 250 pipelines in a newly created Pipeline and then export it. but that is a very tedious work.

how can i export them by Selecting those pipelines and creating their arm template and then by selecting them to delete them in one click

can anyone help me out on this ?

Answer 1

Hello Zouraiz Butt,

As Samy mentioned, you can use the PowerShell script to export pipelines as required by you programmatically.

Please follow the steps below.

• Use PowerShell to get a list of all the pipelines in your Data Factory.
• Filter the list to include only the pipelines you want to export and delete.
• For each pipeline in the filtered list, use PowerShell to export the ARM template to a backup folder.
• Once all the ARM templates have been exported, use PowerShell to delete the pipelines from your Data Factory.

# Connect to your Azure account
Connect-AzAccount


$factoryName = "MyDataFactory"

# Get a list of all the pipelines in your Data Factory
$pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $factoryName

# Filter the list to include only the pipelines you want to export and delete
$backupPipelines = $pipelines | Where-Object { $_.Name -like "*" }

# Export the ARM templates for each pipeline in the filtered list
foreach ($pipeline in $backupPipelines) {
    $template = Get-AzDataFactoryV2Pipeline -DataFactoryName $factoryName -ResourceGroupName $pipeline.ResourceGroupName -Name $pipeline.Name | ConvertTo-Json -Depth 100
    $template | Out-File "C:\Backup\$($pipeline.Name).json"
}

#delete 
foreach ($pipeline in $backupPipelines) {
    Remove-AzDataFactoryV2Pipeline -DataFactoryName $factoryName -ResourceGroupName $pipeline.ResourceGroupName -Name $pipeline.Name
}

This script gets a list of all the pipelines in your Data Factory

You can filter the list of pipelines Where-Object { $_.Name -like "pipeline1" } section

So this will export pipeline1 to C:\backup folder '

Remove-AzDataFactoryV2Pipeline removes the pipeline1 from the datafactory.

I have executed the script and successfully delete the pipeline1 from my data factory. Please test the script and let me know if you run into any issues.

I hope this helps.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.