how to use parameter configuration file to add variables to a armTemplate

Question

how to use parameter configuration file to add variables to a armTemplate

stejpal 0

This question is about ADF deployments, using the dev environment codebase and deploy it to other envs. I would like to use the arm-template-parameters-definition file and instead of just parameterizing property-values as separate parameters in the arm-template. I would like to use the parameter-definition file to generate a set of variables and use those throughout the armTemplate. My Current requirement is to use variable for the retry-value for a web activity. I could not find a whole lot of details on what is and isn't possible with the parameters-file

1 answer

Your answer

Answer 1

QuantumCache 20,371 Moderator

Hello @stejpal Thanks for reaching out on this forum.

I was trying to understand the requirement! Correct me if needed!

I would like to use the parameter-definition file to generate a set of variables....

Did you consider a variable in the parameter file and use it in the ARM template?

Something like below? how does your current ARM templates look like, please add some example on current and expected requirement!!

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "retryValue": {
            "type": "int",
            "defaultValue": 3,
            "metadata": {
                "description": "The number of times to retry the web activity."
            }
        }
    }
}
In the ARM template, you can reference the variable using the parameters function. Here is an example of how to use the variable in the ARM template1:

{
    "name": "WebActivity",
    "type": "Microsoft.DataFactory/factories/pipelines/activities",
    "dependsOn": [
        "[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"
    ],
    "properties": {
        "type": "WebActivity",
        "typeProperties": {
            "url": "https://www.example.com",
            "method": "GET",
            "retry": {
                "count": "@parameters('retryValue')"
            }
        }
    }
}

stejpal 0

Thanks Satish,

Yes, this is regarding data-factory armTemplate generation. I am looking to have the arm-template look exactly like you have shown. I was trying to figure out how to set this behavior using the parameter definition file. Currently I parameterize specific properties using the parameter definition file like shown below but in order to use standard parameter values like you have shown. I need to figure out a way to include those settings in the parameter-definition-file. So that we can build the armTemplate.

{
    "Microsoft.DataFactory/factories": {
        "properties": {
            "globalParameters": {
                "*": {
                    "value": "="
                }
            },
            "globalConfigurations": {
                "*": "="
            },
            "encryption": {
                "*": "=",
                "identity": {
                    "*": "="
                }
            }
        },
        "location": "=",
        "identity": {
            "type": "=",
            "userAssignedIdentities": "="
        }
    },
    "Microsoft.DataFactory/factories/pipelines": {},
    "Microsoft.DataFactory/factories/integrationRuntimes": {
        "properties": {
            "typeProperties": {
                "ssisProperties": {
                    "catalogInfo": {
                        "catalogServerEndpoint": "=",
                        "catalogAdminUserName": "=",
                        "catalogAdminPassword": {
                            "value": "-::secureString"
                        }
                    },
                    "customSetupScriptProperties": {
                        "sasToken": {
                            "value": "-::secureString"
                        }
                    }
                },
                "linkedInfo": {
                    "key": {
                        "value": "-::secureString"
                    },
                    "resourceId": "="
                },
                "computeProperties": {
                    "dataFlowProperties": {
                        "externalComputeInfo": [
                            {
                                "accessToken": "-::secureString"
                            }
                        ]
                    }
                }
            }
        }
    },
    "Microsoft.DataFactory/factories/triggers": {
        "properties": {
            "pipelines": [
                {
                    "parameters": {
                        "*": "="
                    }
                },
                "pipelineReference.referenceName"
            ],
            "pipeline": {
                "parameters": {
                    "*": "="
                }
            },
            "typeProperties": {
                "scope": "="
            }
        }
    },
    "Microsoft.DataFactory/factories/linkedServices": {
        "*": {
            "properties": {
                "typeProperties": {
                    "accountName": "=",
                    "accountEndpoint": "=",
                    "username": "=",
                    "userName": "=",
                    "accessKeyId": "=",
                    "endpoint": "=",
                    "servicePrincipalId": "=",
                    "userId": "=",
                    "host": "=",
                    "clientId": "=",
                    "existingClusterId": "=",
                    "clusterUserName": "=",
                    "clusterSshUserName": "=",
                    "hostSubscriptionId": "=",
                    "clusterResourceGroup": "=",
                    "subscriptionId": "=",
                    "resourceGroupName": "=",
                    "tenant": "=",
                    "dataLakeStoreUri": "=",
                    "baseUrl": "=",
                    "database": "=",
                    "serviceEndpoint": "=",
                    "batchUri": "=",
                    "poolName": "=",
                    "databaseName": "=",
                    "systemNumber": "=",
                    "server": "=",
                    "url": "=",
                    "functionAppUrl": "=",
                    "environmentUrl": "=",
                    "aadResourceId": "=",
                    "sasUri": "|:-sasUri:secureString",
                    "sasToken": "|",
                    "connectionString": "|:-connectionString:secureString",
                    "hostKeyFingerprint": "=",
                    "mlWorkspaceName": "="
                },
                "parameters": {
                    "*": {
                        "defaultValue": "="
                    }
                }
            }
        },
        "Odbc": {
            "properties": {
                "typeProperties": {
                    "userName": "=",
                    "connectionString": {
                        "secretName": "="
                    }
                }
            }
        }
    },
    "Microsoft.DataFactory/factories/datasets": {
        "*": {
            "properties": {
                "typeProperties": {
                    "folderPath": "=",
                    "fileName": "="
                }
            }
        }
    },
    "Microsoft.DataFactory/factories/credentials": {
        "*": {
            "properties": {
                "typeProperties": {
                    "token": "="
                }
            }
        }
    },
    "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints": {
        "properties": {
            "privateLinkResourceId": "=",
            "groupId": "=",
            "fqdns": "="
        }
    },
    "Microsoft.DataFactory/factories/globalparameters": {
        "properties": {
            "*": {
                "value": "="
            }
        }
    }
}

QuantumCache 20,371 Moderator

Sure, here is an example of an ARM template and its corresponding parameter file for an Azure Data Factory pipeline that uses a variable for the retry value of a web activity:

arm-template.json:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "pipelineName": {
            "type": "string",
            "metadata": {
                "description": "Name of the pipeline to create."
            }
        },
        "retryCount": {
            "type": "int",
            "metadata": {
                "description": "Number of times to retry the web activity."
            }
        }
    },
    "variables": {
        "webActivityRetry": "[repeat(activity('WebActivity').retry, parameters('retryCount'))]"
    },
    "resources": [
        {
            "type": "Microsoft.DataFactory/factories/pipelines",
            "apiVersion": "2018-06-01",
            "name": "[parameters('pipelineName')]",
            "properties": {
                "activities": [
                    {
                        "name": "WebActivity",
                        "type": "WebActivity",
                        "dependsOn": [],
                        "policy": {
                            "retry": "[parameters('retryCount')]"
                        },
                        "typeProperties": {
                            "url": "https://www.example.com",
                            "method": "GET"
                        }
                    }
                ],
                "variables": {
                    "webActivityRetry": {
                        "type": "String",
                        "value": "[variables('webActivityRetry')]"
                    }
                }
            }
        }
    ]
}

arm-template-parameters.json:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "pipelineName": {
            "value": "MyPipeline"
        },
        "retryCount": {
            "value": 3
        }
    }
}

# File: azure-pipelines.yml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: AzureResourceGroupDeployment@2
  inputs:
    azureSubscription: 'MyAzureSubscription'
    resourceGroupName: 'MyResourceGroup'
    location: 'eastus'
    csmFile: 'arm-template.json'
    csmParametersFile: 'arm-template-parameters.json'
    overrideParameters: '-webActivityRetry 5'

If the response is helpful, please click "Accept Answer" and upvote it. So that we can close this thread.

stejpal 0

Thanks Sorry I have not been very clear with my request. We have a github repository for our data-factory instance which consists of multiple folders for each type of object like linkedservice, trigger, dataset. which in turn have multiple json files under each folder. when we want to publish our data-factory main branch to our data-factory instance we run a github workflow which executes a github-action. In this case it is Azure/data-factory-export-action. This action uses the parameters definition file to generate the armTemplate. Is there a way to update the parameter definition file to populate the armTemplate as I do not want to script this out using powershell or hand craft the armTemplate.

    - name: Export ARM Template
      id: export
      uses: Azure/data-factory-export-action@v1.1.0
    - name: update step summary
      run:  echo "[x] _Vailidate and build template_" >> $GITHUB_STEP_SUMMARY 
    - name: Print output
      id: print
      run: |
            echo "######################################################"            
            echo  ${{ steps.export.outputs.arm-template-directory }}
            echo ${{ env.uri }}
            echo "######################################################"
            ls -R ${{ steps.export.outputs.arm-template-directory }}
            echo "- storage account location=${{env.directory}}" >> $GITHUB_STEP_SUMMARY
            echo "- deployment name=${{env.deployment}}" >> $GITHUB_STEP_SUMMARY
            echo "- armTemplate location=${{env.uri}}" >> $GITHUB_STEP_SUMMARY
            echo "  " >> $GITHUB_STEP_SUMMARY
            echo "[x] _List armTemplate files_ " >> $GITHUB_STEP_SUMMARY

QuantumCache 20,371 Reputation points Moderator

2023-06-21T00:35:48.4466667+00:00
You can try update the parameter values in the parameter definition file to populate the ARM template with the desired values.

Open the parameter definition file in a text editor.

Locate the parameter that you want to update.

Update the value of the parameter to the desired value.

Save the parameter definition file.

When you run the GitHub workflow, the Azure/data-factory-export-action will use the updated parameter definition file to generate the ARM template with the updated parameter values.

Note that the parameter definition file is specific to the ARM template that you are generating. If you have multiple ARM templates, you will need to create a separate parameter definition file for each ARM template.

I hope this helps! Let me know if you have any further questions.

stejpal 0

Thanks Satish, I have updated the parameter-definition-file in the past to parameterize some of the properties. I was looking to use variables or parameters in the parameter-definition file so that the generated armTemplate uses variables or parameters instead of being parameterized. Here is currently what I am doing to select the retry property in the parameter-definition file.

    "Microsoft.DataFactory/factories/pipelines": {
        "properties": {
            "activities": [
                {
                    "type": "WebActivity",
                    "policy": {
                        "retry": "="
                    }
                }
            ]
        }
    }

This generates an armTemplate with parameters for each retry property for the webActivity. Like so

        "PLSPCampaignExtract_properties_30_policy_retry": {
            "type": "int",
            "defaultValue": 0
        },
        "PLSPCampaignExtract_properties_31_policy_retry": {
            "type": "int",
            "defaultValue": 0
        }

I would like it to mimic what you had shown. Where all the properties reference the same parameter Can we do something similar to this in the parameter file

    "parameters": {
        "defaultretry": {
            "defaultValue": 2,
            "type": "int"
        }
    },
"Microsoft.DataFactory/factories/pipelines": {
        "properties": {
            "activities": [
                {
                    "type": "WebActivity",
                    "policy": {
                        "retry": "[concat(parameters('defaultretry')]"
                    }
                }
            ]
        }
    }

Share via

how to use parameter configuration file to add variables to a armTemplate

1 answer

Your answer