Exporting Selected Azure Data Factory Pipelines

Zouraiz Butt 0 Reputation points
2023-05-02T12:15:15.3833333+00:00

Hi,

I have a question, I have Data Factory which has almost 850 pipelines.

Out of 850 pipelines. 250 of them are created using as Back up or created for versioning of pipelines.

now i want to take backup of those pipelines and want to store their ARM template in backup folder and delete them for ADF
one solution I have in mind is that i create a pipeline and put all 250 pipelines in a newly created Pipeline and then export it. but that is a very tedious work.

how can i export them by Selecting those pipelines and creating their arm template and then by selecting them to delete them in one click

can anyone help me out on this ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,948 questions
{count} votes

1 answer

Sort by: Most helpful
  1. BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
    2023-05-03T23:29:53.21+00:00

    Hello Zouraiz Butt,

    As Samy mentioned, you can use the PowerShell script to export pipelines as required by you programmatically.

    Please follow the steps below.

    • Use PowerShell to get a list of all the pipelines in your Data Factory.
    • Filter the list to include only the pipelines you want to export and delete.
    • For each pipeline in the filtered list, use PowerShell to export the ARM template to a backup folder.
    • Once all the ARM templates have been exported, use PowerShell to delete the pipelines from your Data Factory.
    # Connect to your Azure account
    Connect-AzAccount
    
    
    $factoryName = "MyDataFactory"
    
    # Get a list of all the pipelines in your Data Factory
    $pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $factoryName
    
    # Filter the list to include only the pipelines you want to export and delete
    $backupPipelines = $pipelines | Where-Object { $_.Name -like "*" }
    
    # Export the ARM templates for each pipeline in the filtered list
    foreach ($pipeline in $backupPipelines) {
        $template = Get-AzDataFactoryV2Pipeline -DataFactoryName $factoryName -ResourceGroupName $pipeline.ResourceGroupName -Name $pipeline.Name | ConvertTo-Json -Depth 100
        $template | Out-File "C:\Backup\$($pipeline.Name).json"
    }
    
    #delete 
    foreach ($pipeline in $backupPipelines) {
        Remove-AzDataFactoryV2Pipeline -DataFactoryName $factoryName -ResourceGroupName $pipeline.ResourceGroupName -Name $pipeline.Name
    }
    

    This script gets a list of all the pipelines in your Data Factory

    You can filter the list of pipelines Where-Object { $_.Name -like "pipeline1" } section

    So this will export pipeline1 to C:\backup folder '

    Remove-AzDataFactoryV2Pipeline removes the pipeline1 from the datafactory.

    I have executed the script and successfully delete the pipeline1 from my data factory. Please test the script and let me know if you run into any issues.

    User's image

    I hope this helps.

    If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.