Feature Request: prevent global parameters in git collaboration branch from being published to live mode

asked 2022-06-10T06:48:05.757+00:00
Michael Van de Borne 1 Reputation point

Our global parameters between differents ADF environments are managed using Terraform. As such, we'd like to prevent (using a checkbox) that the global parameters present in the git collaboration branch would make it to the live mode when the Publish button is pressed, hence preventing global parameters (managed by Terraform) from getting overwritten.

Managing Global Parameters with Terraform is very convenient, because it is a perfect tool to handle environment specific settings (address of the key vault, Databricks workspace URL, Databricks cluster ID, ...). But at the moment, when a new global parameter is set with Terraform (directly in live mode), Data Factory detects a discrepancy between the git collaboration branch and the live mode when publishing, and overwrites the parameters managed by Terraform.

The alternative is to handle all the global parameters in all environments manually, but this process is error-prone and lacks automation.

Also note that the current "Include in ARM template" checkbox has nothing to do with this feature request (as already clarified with Azure support), as it aims to prevent deploying the global parameters to the next environment when using a CICD pipeline to promote ADF content, but not to prevent global parameters in the git collaboration branch from getting published to live mode.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
No comments
{count} votes

1 answer

Sort by: Most helpful
  1. answered 2022-06-13T11:18:10.057+00:00
    AnnuKumari-MSFT 19,116 Reputation points Microsoft Employee

    Hi @Michael Van de Borne ,
    Welcome to Microsoft Q&A platform and thanks for posting your query.
    As I understand your ask here, you are trying to find a workaround to deploy global parameters in ADF from dev environment to higher environments.

    You can leverage Powershell script to achieve this requirement. If you keep the 'Include in ARM template' box unchecked and publish the ADF changes, or export an ARM template with global parameters, a folder called globalParameters is created with a file called your-factory-name_GlobalParameters.json.

    You need to download this .json file in the local and provide the same while running the below script . Provide details like subscription id, and adf name of the higher environment and you should see the Global parameters reflecting in higher env.

    Kindly check this doc for more details: Deploying using PowerShell

    param  
    (  
        [parameter(Mandatory = $true)] [String] $globalParametersFilePath,  
        [parameter(Mandatory = $true)] [String] $resourceGroupName,  
        [parameter(Mandatory = $true)] [String] $dataFactoryName  
    )  
      
    Import-Module Az.DataFactory  
      
    $newGlobalParameters = New-Object 'system.collections.generic.dictionary[string,Microsoft.Azure.Management.DataFactory.Models.GlobalParameterSpecification]'  
      
    Write-Host "Getting global parameters JSON from: " $globalParametersFilePath  
    $globalParametersJson = Get-Content $globalParametersFilePath  
      
    Write-Host "Parsing JSON..."  
    $globalParametersObject = [Newtonsoft.Json.Linq.JObject]::Parse($globalParametersJson)  
      
    # $gp in $factoryFileObject.properties.globalParameters.GetEnumerator())   
    # may  be used in case you use non-standard location for global parameters. It is not recommended.   
    foreach ($gp in $globalParametersObject.GetEnumerator()) {  
        Write-Host "Adding global parameter:" $gp.Key  
        $globalParameterValue = $gp.Value.ToObject([Microsoft.Azure.Management.DataFactory.Models.GlobalParameterSpecification])  
        $newGlobalParameters.Add($gp.Key, $globalParameterValue)  
    }  
      
    $dataFactory = Get-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $dataFactoryName  
    $dataFactory.GlobalParameters = $newGlobalParameters  
      
    Write-Host "Updating" $newGlobalParameters.Count "global parameters."  
      
    Set-AzDataFactoryV2 -InputObject $dataFactory -Force  
    

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators