How to overrideARMparameter - dependsOnproperty of synapse pipeline?

Preetha Rajesh 126 Reputation points
2023-07-08T11:40:39.05+00:00

Requirement is to override the property of synapse pipeline->dependsOn property, while deploying to higher environment.

The steps followed by me are->

  1. Updated template-parameters-definition.json file to make the dependsOn property of Microsoft.synapse.pipeline
    "dependsOn: "-:-dependsonpipeline:array"
  2. Once the code is published from master branch of synapse repo, I got the additional parameter in templateParametersforWorkspace.json file. All good so far.
  3. Next step , I wanted to override this parameter in the YAML pipeline, by giving appropriate variable and calling that variable in the OverrideARMparameters section.

Variables:
-name: pipelinedependsOn
-values : I tried various option to pass the set of depndsOn values(in my case, the pipeline depends on dataflow,linked service and sparkpool.The sparkpool name is different for different environment; thats why I am trying to override the depensOn property for a particular pipeline)

I am getting major below two errorswhen I run the YAML pipeline-

  1. Failed to parse package syntax error unexpected token
  2. Failed to parse typeError: artifacts.dependOn.oreach is not a function

I know I am doing some mistake while passing the variable value and overrideARMparameters value or depends (array type). Can someone please help on this?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,186 questions
0 comments No comments
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 34,011 Reputation points Microsoft Employee
    2023-08-02T10:04:18.4033333+00:00

    Hi Rui Cunha ,

    Thankyou for putting effort and sharing the approach for resolving the issue.

    Preetha Rajesh ,

    Glad that you figured out the way to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

    YAML version needs to be compatible with the spark config. By upgrading the version and changing the task from "synapse deployment task@0" to "synapse deployment task@2" , it resolved the issue and the deployment was successful.

    Kindly accept the answer by clicking on Accept answer button so that the solution reaches the community in case anyone is facing similar issue. Thankyou.


2 additional answers

Sort by: Most helpful
  1. AnnuKumari-MSFT 34,011 Reputation points Microsoft Employee
    2023-07-11T05:51:53.7566667+00:00

    Hi Preetha Rajesh ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

    It seems you are trying to override the dependsOn property of a Synapse pipeline while deploying to a higher environment.

    If you need to override the artifact properties that are not parameterized by the default parameters template(TemplateParametersforworkspace.json), then you need to use "custom parameters template".

    To override the default parameter template, create a custom parameter template named template-parameters-definition.json in the root folder of your Git branch. You must use this exact file name.

    When Azure Synapse workspace publishes from the collaboration branch or the deployment task validates the artifacts in other branches, it reads this file and uses its configuration to generate the parameters. If Azure Synapse workspace doesn't find that file, it uses the default parameter template.

    Ex: The below code in the custom template-parameters-defintion.json will parameterize all parameters at the pipeline level in the Synapse Deployment task.

    "Microsoft.Synapse/workspaces/pipelines": {  
     "properties": {  
                "parameters":{  
                        "*": {"*":"="}  
                     
                    }
    

    Reference documents:

    https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/cicd-automation-in-synapse-analytics-taking-advantage-of-custom/ba-p/3513546

    https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#create-custom-parameters-in-the-workspace-template

    Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou


  2. Rui Cunha 0 Reputation points Microsoft Employee
    2023-08-01T11:20:49.15+00:00

    the dependsOn is not meant to be explicitly parameterized in your custom parameter template.

    instead, you should parameterize the corresponding property, in this case the spark pool reference name.

    by doing this, the corresponding dependsOn entry will be implicitly parameterized as well. From the moment you override the parameter, the dependency will be implicity overridden as well.

    Here's a sample code demonstrating how you can parameterize the spark pool name and from the attached screenshot below you will see the parameterization also being applied to the dependsOn property.

      "Microsoft.Synapse/workspaces/pipelines": {
            "properties": {
                "activities": [{
                    "typeProperties": {
                        "sparkPool": {
                            "referenceName":"="
                        }
                    }
                }]
            }
        },
    

    User's image


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.