Azure Synapse CI/CD Pipeline - Override Apache Spark Pool Name in Notebooks

Question

I'm working on setting up a CI/CD pipeline for an Azure Synapse workspace, and I need to create custom parameters in the deployment process to handle environment-specific values. Specifically, I need to override the Apache Spark pool name used in the development environment (synDEVSPark) to the production Apache Spark pool name (synPROSPark1).

I've followed the instructions from the Microsoft documentation, but I'm encountering an error during the synapse deployment task.

Here are the steps I've taken:

1. Created a template-parameters-definition.json file to define the parameters (shown in the image below).
2. Set up the override parameter in the Synapse deployment task (shown in the image below:
3. Created a pipeline variable to hold the production Apache Spark pool name (shown in the below image). Despite these efforts, the deployment fails with the following error (shown in the image below): Failed to fetch the deployment status {"code":"400","message":"CreateOrUpdateNotebook failed: [statusCode from ADF:BadRequest, ErrorMessage:{\"code\":\"BadRequest\",\"message\":\"The document creation or update failed because of invalid reference 'synDEVSPark'... How can I correctly override the Apache Spark pool name in the Synapse deployment task to ensure it uses the production Spark pool (synPROSPark1) instead of the development Spark pool (synDEVSPark)? Is there a specific way to reference the Apache Spark pool in the parameters, considering it is not a linked service? Any help or guidance on what I might be doing wrong or how to properly configure the override parameters would be greatly appreciated. Thank you!

Answer 1

Hi @Tsu Kernik

Thanks for the question and using MS Q&A platform.

Please correct me if my understanding is wrong. You are looking to parameterize the Notebook parameter, to replace your Spark pool name with a different value (without the default value) when deploying to a higher environment. You can use the below code in your template-parameter-definition file. This code will expose the notebook parameters.

"Microsoft.Synapse/workspaces/notebooks": {

        "properties": {

            "bigDataPool": {

                "referenceName": "="

            },

             "metadata": {

                "a365ComputeOptions": {

                        "id": "=",

                         "name": "=",

                        "endpoint": "="

                }

            }

        }

     }

on the next step, go to the Workspace Deployment task in your Release Pipeline and add these new parameters in the “OverrideParameters” section. Here I have added Notebook 1_properties_bigDataPool_referenceName = DemoUAT ( to change the sparkpool name from demo to DemoUAT)

I hope this helps. Please let me know if you have any further questions.