I try to use DevOps to deploy ADF code from DEV to SIT, When I do the stress testing I found there're ARM template limits.

Question

Cause our team got a large number of pipelines and dataflow in Data Factory. our arm template size for deployment already beyond 4MB.
I also got suggestions to using linked Template, however, it requires an additional storage account to store files. Originally, all my files are stored in repos. Our team also wants to use it to implement ci/cd. In this way, I don't think Devops has reduced our expenses and improved our efficiency and simplified deployment process . I will be very grateful if anyone have advice about how to solve large size data factory deploy.

Answer 1

Hi @Anonymous ,

Welcome to Microsoft Q&A platform and thankyou for posting your question.

As I understand your issue, you are trying to make use of CICD process in order to deploy the ADF changes from DEV to higher environments. However, the ARM template size is crossing 4mb and you are facing limit exceeded error. Please let me know if my understanding is incorrect.

You can leverage this article to Resolve errors for job size exceeded
As you rightly called out , you can divide your template into logical groups and add a linked template for each group. Reference: Using linked and nested templates when deploying Azure resources

The below mentioned articles provides step by step guidance on :

1. How to Deploy a local ARM template
2. How to Deploy a linked template
3. How to create CICD pipeline to deploy the ARM templates

In the above tutorials, it's mentioned that we need to store the linked template in a storage account, and secure the linked template by using SAS token.

Also, I would like to highlight this solution : Reduce max size .

You can also try to store all the pipeline parameters in a control Table instead of having them in the pipeline itself and Create a master pipeline over your pipelines having large number of parameters which would have lookup activity fetching the parameters from the table and then use execute pipeline activity to invoke the actual pipelines and pass the parameters from lookup activity output to the child pipeline . This will reduce the number of parameters referenced in ARM template.

For queries more specific to Azure DevOps, you can post it on this platform : https://developercommunity.visualstudio.com/spaces/21/index.html
The Azure DevOps team and community are active and answering questions there.

You can also ask for help : https://stackoverflow.com/questions/tagged/azure-devops.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators