How to keep ARM Template in sync when modifying Azure Data Factory?

Question

I'm unclear on the process for maintaining the ARM Template for deployments when modifying an Azure Data Factory (ADF). I see two options:

1. Modify the ADF in the browser-based drag-n-drop editor and use the generated Json ARM Template as is.
2. Manually maintain the ARM in either Json or Bicep.

Option 1 is not good because the generated ARM has hard-coded values (ex. Blob Event Triggers) that make deployment to different environments cumbersome.

Option 2 has the obvious overhead of having to keep the ARM in sync with the ADF.

Is there another option or a workaround for these problems?

Answer 1

You can modify the ARM templates to use parameters for environment-specific values instead of hard-coded constants. This way, you can use the same ARM template across different environments by changing the parameters file according to the environment (DEV, UAT, PROD).

You can set up a CI/CD pipeline using Azure DevOps or GitHub Actions to automate the synchronization process.

My idea is you start by extracting the ARM template from the ADF instance after changes are made using the Azure Resource Manager.

Then you parametrize the template by replacing the environment-specific values with parameters.

Check the updated ARM template back into your source control system.

And finally, deploy the ARM template across various environments using different parameter files.

For Bicep, it supports modularization and parametrization, which helps you define resources in a cleaner, more reusable way. It also compiles down to ARM templates.

Always store your ARM templates or Bicep files in a version control system to track changes, manage versions, and revert to previous versions if something goes wrong.

If you are comfortable with ARM template toolkit, I recommend it : https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/test-toolkit

This toolkit can be used in your CI/CD pipeline to test the ARM templates for best practices and compliance before deployment.

Answer 2

Hello Moyer, Todd,

Welcome to the Microsoft Q&A forum.

Here are some strategies that you can follow.

Parameterization and Variables: Utilizing parameters and variables within ARM templates provides the flexibility needed for managing deployments across different environments. Parameters allow you to input environment-specific values at deployment time, while variables can be used to simplify complex expressions within the template.

https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/best-practices

CI/CD Pipeline:Implementing CI/CD with ARM templates can help automate the deployment and synchronization process. Azure Pipelines can deploy these ARM templates to different environments, and you can use Git/ADO integration for source control to manage changes. This approach includes developing and debugging changes in a separate feature branch, creating pull requests for code review, and merging changes for automatic deployment.

https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery

Pre- and post-deployment script. Before the Resource Manager deployment step in CI/CD, you need to complete certain tasks, like stopping and restarting triggers and performing cleanup.

https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-sample-script

https://github.com/Azure/Azure-DataFactory/blob/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1

I hope this helps. Please let me know if you have any further questions.