ERROR: {"code": "InvalidTemplate", "message": "Deployment template validation failed}

Question

Hi Team,

We are trying to deploy to ADF instance using Jenkins as CI/CD platform. We are following the documentation for reference: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements#continuous-deployment-improvements

We used the below command:

az deployment group create --name ${name}-${version} --resource-group ${azureDataFactoryResourceGroup} --parameters @arm_template_parameters.json --mode Incremental --template-uri 'https://${storageAccountName}.blob.core.windows.net/${name}/${environment}/${version}/ArmTemplate_master.json' --query-string '${sasToken}' --parameters containerUri=https://${storageAccountName}.blob.core.windows.net/${name}/${environment}/${version} --parameters containerSasToken='?${sasToken}'"

But we got the following error:

ERROR: {"code": "Invalid Template", "message": "Deployment template validation failed: 'The template parameters 'ls_xxx_properties_typeProperties_url, ls_xxxx_properties_typeProperties_serviceEndpoint, trg_xxx_Coverage_properties_xx_xx_xx_parameters_SourceName' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time.

We identified that the above parameters are not present in the Source ADF ArmTemplateParameters_master.json file but they are present in the Target ADF ArmTemplateParameters_master.json file. We would like to understand below :

1. What can we do to avoid this error?
2. If the parameters not present in Source Arm template file will the pipeline fail for all deployments?
3. Do we need to have same linked services across all ADF environments?

Request your inputs and help.
Thanks,

Answer 1

@ShaikNisharunnisa-3795 You're on the right track! Here's a breakdown of your questions regarding deploying pipelines and linked services:

Copying Pipelines with ARM Templates:

Removing Unused Parameters:

• Yes, you can remove parameters from arm_templates_parameter.json (DEV environment) that are not present in ArmTemplate_master.json (SIT environment).
• This won't cause issues as long as the removed parameters are not used in the actual template logic.

Default Values:

• Removing unused parameters won't affect default values in the SIT environment. Default values are defined within the ARM template itself (not the parameter file).
• However, if a parameter in arm_templates_parameter.json has a specific value set (different from the SIT default), removing it will cause the deployment to use the SIT default value.

Deploying Across Environments (SIT to UAT):

Replacing Values:

• Deploying the SIT template to UAT won't automatically replace UAT-specific values. You'll need to provide the relevant UAT values through the arm_templates_parameter.json file for the UAT deployment.

Linked Services:

You don't necessarily need identical linked services across all environments (DEV, SIT, UAT).

However, the ARM template might reference specific linked services through parameters. In that case:

• Ensure the referenced linked services exist in the target environment (UAT).
• If they don't exist, create them in UAT or modify the template to use linked services already present in UAT.
• Update the arm_templates_parameter.json for UAT to reference the appropriate linked services.

Avoiding Deployment Errors:

By following these guidelines, you can avoid the "Invalid Template" error:

• Keep arm_templates_parameter.json (DEV or UAT) consistent with the parameters defined in the SIT template (ArmTemplate_master.json).
• If a parameter is referenced in the DEV/UAT parameter file but not used in the template itself, remove it from the parameter file.
• Ensure referenced linked services exist in the target environment or update the template to use existing ones.

Additional Tips:

• Consider using environment-specific parameter files for DEV, SIT, and UAT to manage configuration differences easily.
• Use Azure DevOps variables or Azure Key Vault to store sensitive information like connection strings instead of hardcoding them in the parameter files.

 

Answer 2

Hi Tech_Savy_Nisha,

Thanks for reaching out to Microsoft Q&A.

What can we do to avoid this error?

Some parameters specified in the parameters file are not present in the original template, hence causing this error. Review the parameters & validate the template to ensure formatting and typos are not present.

Try setting the pipeline to allow defaults for missing parameters, it can use those values when a parameter is missing in the template. Though this might help the pipeline to not fail but the catch is the behavior might change depending upon the default parameter. Also, explore the optional and dynamic parameter options if they can suit your scenario.

If the parameters not present in Source Arm template file will the pipeline fail for all deployments?

Not necessarily, it depends. If the pipeline is configured with strict validation with the requirement for all parameters to be present in the template, then any missing parameters will cause the deployment to fail. In such cases, the pipeline will fail for all deployments until the missing parameters are fulfilled.

Do we need to have same linked services across all ADF environments?

Dont think it is required and is possible to maintain the same LS across envs.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.