Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Need help on template-parameters-definition.json to override the parameters in notebook present in synapse pipeline
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 75%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAN%3C/text%3E%3C/svg%3E)
ABDUL NAYEEM
20
Reputation points
Hello,
There is pipeline in azure synapse workspace which consists of notebooks which are having parameters of spark pools attached to them. I have added template-parameters-definition.json in main branch which consists of below code.
{
"Microsoft.Synapse/workspaces/notebooks": {
"properties": {
"bigDataPool": {
"referenceName": "=:-name"
},
"metadata": {
"a365ComputeOptions": {
"id": "=:-id",
"endpoint": "=:-endpoint"
}
}
}
},
"Microsoft.Synapse/workspaces/pipelines": {
"properties": {
"parameters":{
"*": {"*":"="}
},
"bigDataPool": {
"sparkPoolName": {
"referenceName": "=:-name"
}
}
}
}
}
But the parameters passed inside the notebook is still not getting reflected in template-parameters-definition.json ,because of which I am not able to override the parameters passed inside the notebook.
Looking forward for response,
ABD.
Azure Synapse Analytics
{count} votes
-
Zeeshan Nasir Bajwa 691 Reputation points2023-04-17T06:33:29.7733333+00:00 Hello ABD, It seems like you are trying to override the parameters of a notebook within an Azure Synapse pipeline using a template-parameters-definition.json file. However, the parameters passed inside the notebook are not getting reflected in the JSON file. To override the parameters passed inside the notebook, you need to ensure that the parameters in the JSON file have the same name as the parameters in the notebook. The JSON file should also define the parameter type and default value, if applicable. Here's an example of how you can define a parameter in the JSON file:
"Microsoft.Synapse/workspaces/notebooks": { "properties": { "parameters": { "myParam": { "type": "String", "defaultValue": "default_value" } } } } }In the above example, the parameter "myParam" is defined with a type of "String" and a default value of "default_value". You can replace the default value with your own value, or leave it as is if the default value is suitable for your needs. Make sure that you define all the necessary parameters in the JSON file, and that the parameter names match exactly with the parameter names in the notebook. Once you have defined the parameters in the JSON file, you should be able to override the values passed in the notebook. I hope this helps! Let me know if you have any further questions.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator2023-04-19T22:05:44.6066667+00:00 Hello ABD NAY, Welcome to Microsoft Q&A platform. Please correct me if my understanding is wrong. You are looking to parameterize the Notebook parameter, to replace your Spark pool name with a different value (without the default value) when deploying to a higher environment. You can use the below code in your template-parameter-definition file. This code will expose the notebook parameters.
"Microsoft.Synapse/workspaces/notebooks": { "properties": { "bigDataPool": { "referenceName": "=" }, "metadata": { "a365ComputeOptions": { "id": "=", "name": "=", "endpoint": "=" } } } }on the next step, go to the Workspace Deployment task in your Release Pipeline and add these new parameters in the “OverrideParameters” section. Here I have added Notebook 1_properties_bigDataPool_referenceName = DemoUAT ( to change the sparkpool name from demo to DemoUAT)
I hope this helps. Please let me know if you have any further questions.
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2023-04-19T22:06:09.7233333+00:00 Hello ABD NAY, I have updated the code in my previous comment. Can you please follow this and let me know if you have any further questions.
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2023-04-20T15:52:03.7633333+00:00 Hello ABD NAY, I am checking to see if you have tried the above solution. Please let me know if you have any further questions.
-
ABDUL NAYEEM 20 Reputation points
2023-04-20T20:32:14.7966667+00:00 Hello @Bhargava-MSFT ,
Thanks for the response !!
Actually parameterization of notebooks is done and its working fine from the code I have added to the template-parameters-definition.json , but my question was related to synapse pipeline where as this pipeline consists of notebooks connected to each other yet they are attached to respective spark pools , when I am running CI/CD pipeline it is failing to deploy that synapse pipeline until I detach or remove that spark pools attached to the respective notebooks in the synapse pipeline and then I was able to deploy synapse pipeline to higher env . So what I was looking for is...what should be added template-parameters-definition.json same like for note books so that the artifact will have parameters and can be override in CI./CD pipeline and deploy this synapse pipeline to other or higher environment . -
ABDUL NAYEEM 20 Reputation points
2023-04-20T20:49:10.3+00:00 Hello @Bhargava-MSFT
Thanks for the response!!Actually parameterize of notebooks is done and its working fine from the code I have added to template-parameters-definition.json but my question was regarding synapse pipeline where in this pipeline notebooks are connected to each other and attached to respective spark pools, so when I am running CICD pipeline, CICD is failing until I detach or remove these spark pools of this synapse pipeline ,so what code should be added to template-parameters-definition.json as like notebooks so that parameters will be generated with default values and can be override with cicd pipeline when deploying to higher environment
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2023-04-21T00:37:21.7933333+00:00 Hello ABD NAY,
Please correct me If my understanding needs to be corrected.
for ex: I have a pipeline name - pipeline1
which has notebook1 attached to demo pool. (this is in Dev)
When you are trying to deploy the pipeline from dev to UAT, your release pipeline is failing if the spark pool is attached to the Notebook1
If you detach the demo pool from the notebook and publish the changes, your release pipeline is success. If this is correct, the reason for the failure is due to different sparkpool name on your UAT environment. You need to have the same sparkpool name on your UAT end. Otherwise, you need to use the custom parameters to parameterize the sparkpool name, and replace the UAT spark pool name on the override parameters template, as shown in the screenshot below.
I hope this helps. Please let me know if you have any further questions.
-
ABDUL NAYEEM 20 Reputation points
2023-04-21T04:45:23.6233333+00:00 Hello @BhargavaGunnam-MSFT Yes, your understanding is correct . Below is the content of template-parameters-definition.json
{ "Microsoft.Synapse/workspaces/notebooks": { "properties": { "bigDataPool": { "referenceName": "=:-name" }, "metadata": { "a365ComputeOptions": { "id": "=:-id", "endpoint": "=:-endpoint" } } } }, "Microsoft.Synapse/workspaces/pipelines": { "properties": { "parameters":{ "*": {"*":"="} } } } }But the parameter is not reflecting in TemplateParametersForWorkspace.json to override it in CICD pipeline**..** Kindly let me know if this template-parameters-definition.json is correct or not Kindly share the content of template-parameters-definition.json . Looking forward for you response..
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2023-04-21T20:01:42.2266667+00:00 Hello ABD NAY, Thank you for the details. I can see the notebook parameters from my end, but when I use the values to deploy, it fails with the below error. For the pipeline parameters, your code will parameterize everything under the properties section. I am checking with my internal team to get more details on it. I will update you as soon as I hear from them.
"Microsoft.Synapse/workspaces/notebooks": { "properties": { "bigDataPool": { "referenceName": "=" }, "metadata": { "a365ComputeOptions": { "id": "=", "name": "=", "endpoint": "=" } } } } -
ABDUL NAYEEM 20 Reputation points
2023-04-24T09:34:22.8066667+00:00 Hello @Bhargava-MSFT , Sure Bhargava , I will be looking for your response. ABD.
-
ABDUL NAYEEM 20 Reputation points
2023-04-26T03:12:43.3366667+00:00 Hello @BhargavaGunnam-MSFT, Have you got any details regarding this. Kindly let me know. Looking forward. ABD.
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2023-04-26T17:26:46.1966667+00:00 Hello ABD NAY,
Thanks for your reply
I reached out to my internal team for assistance on this matter, but unfortunately, I did not receive any response from them. However, I could use the custom code to parameterize one level of the pipeline parameters.
If you have a support plan, I recommend opening a support case for further assistance with this issue. Alternatively, you can create a new thread in our Dedicated Azure DevOps support channel for community support.
Please let me know if you require assistance opening a support case, and I will be happy to provide you with a one-time free support request.
"Microsoft.Synapse/workspaces/pipelines": { "properties": { "activities": { "*": { "*": "=" } } } }
Sign in to comment
Sign in to answer