We are currently creating CI /CD pipeline for ADF in Azure Devops and below are the steps that we are following,
1. Use npm to export the ARM template (Autoate Publish - not from UX) from azure pipelines and create artifact
2. Use the artifact with ARM template to deploy it(release) to other environment like UAT and PRD
We have succeeded in these two, however currently we are integrating ADF with Azure Data bricks. We have created the linked service for the Databricks and couldn't find the required parameters in ARM template for Data bricks to override in release pipelines. But we ultimately found some options from documentation about two approaches about parameterization.
The documentation is explaining about Databricks link using existing cluster id - using "type properties" and to use arm-template-definition.json inside the repo. Issue with this is every time we create a new linked service which requires a parameter to be included in arm-template-definition.json which is a manual task. Is this correct?
Use of global parameters - as per documentation CI /CD in ADF and override can be achieved using global params which is set at factory level. Is there any drawback on this?
Can you please help me to understand which is the best approach as in to avoid any manual intervention and automate ADF - databricks linked services in CI CD.
Also please be noted - we are not going to use existing interactive cluster in Data bricks but create new cluster every time on releases. Give us a pointer if there are any documentation to it.