How to deploy Synapse Workspace Spark Pool to different environment using CI/CD?

Question

Hello, I have issue with deploying Synapse workspace artifact from one evnironment to another with Azure DevOps Pipelines

The workspace I want to re-deploy has some notebooks and Apache Spark pool.

I am using the Synapse workspace deployment@2 with operation validate to download the TemplateForWorkspace.json artifact and then deploy the artifact to another environment with the same task with operation deploy.
The problem is, that when I check the artifact (relevant code snippet below), the notebook RelevantNotebook, that should run on that pool doesn't have it as a dependency (almost last line) and to me it seems that the pool resource (the first one) has very few attributes listed (I am missing e.g. python, spark, Delta lake versions, idle minutes, and so on).

	"resources": [
		...
		{
            "name": "[concat(parameters('workspaceName'), '/synspnbigais')]",
            "type": "Microsoft.Synapse/workspaces/bigDataPools",
            "apiVersion": "2019-06-01-preview",
            "properties": {
                "annotations": []
            },
            "dependsOn": []
        },
        {
            "name": "[concat(parameters('workspaceName'), '/RelevantNotebook')]",
            "type": "Microsoft.Synapse/workspaces/notebooks",
            "apiVersion": "2019-06-01-preview",
            "properties": {
                "nbformat": 4,
                "nbformat_minor": 2,
                "bigDataPool": {
                    "referenceName": "synspnbigais",
                    "type": "BigDataPoolReference"
                },
                "sessionProperties": {
                    "driverMemory": "56g",
                    "driverCores": 8,
                    "executorMemory": "56g",
                    "executorCores": 8,
                    "numExecutors": 2,
                    "conf": {
                        "spark.dynamicAllocation.enabled": "false",
                        "spark.dynamicAllocation.minExecutors": "2",
                        "spark.dynamicAllocation.maxExecutors": "2",
                        "spark.autotune.trackingId": "some-long-id"
                    }
                },
				...	
			},
            "dependsOn": []
		}
	]
}

Answer 1

HI Jan Kučera ,

I understand that you are trying to deploy a Synapse workspace artifact , particularly spark pool from one environment to another using Azure DevOps Pipelines.

You can write a powershell script to replace any parameter like 'dedicated sql pool name' , 'spark pool name' , 'Integration Run time' Connection string and add a 'Run inline Powershell' task in your release pipeline to run the code .

Below is a set of sample code :

1. First code is to replace dedicated sql pool name .
2. Second one is to replace spark pool name.AI ConvertCopy

(Get-Content -path $(System.DefaultWorkingDirectory)/_SynapseRepo/annu-analytics/TemplateForWorkspace.json) -replace 'dev_sql_pool ','prod_sql_pool ' | Set-Content -Path $(System.DefaultWorkingDirectory)/_SynapseRepo/annu-analytics/TemplateForWorkspace.json   
(Get-Content -path $(System.DefaultWorkingDirectory)/_SynapseRepo/annu-analytics/TemplateForWorkspace.json) -replace 'https://annu-analytics.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/devsparkpool','https://annu-analytics.prod.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/prodsparkpool'| Set-Content -Path $(System.DefaultWorkingDirectory)/_SynapseRepo/annu-analytics/TemplateForWorkspace.json   

Here , you need to replace the path where '_SynapseRepo' should be replaced by the artifact name that you provided for release pipeline and 'annu-analytics' should be replaced with your synapse workspace name Reference for artifact name:

Hope it helps. Please do let us know how it goes. Thankyou