how to update Azure Synapse spark pool with an exiting configuration in workspace

Question

I have an existing Synapse configuration that has been set from the Synapse studio. Now I have a new spark pool created and want to bind that configuration to the pool. On the portal, it can be set on the sparkpool. But what is the programmatic way to do that, e.g. azure cli, etc.?

Answer 1

First thing, install the Azure CLI, if you haven't already :

   curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Then, login to it :

   az login

Set the required configuration in a JSON file, for example, create a file named spark-pool-config.json with your desired configuration:

   {

     "name": "your-spark-pool-name",

     "tags": {},

     "autoScale": {

       "enabled": true,

       "minNodeCount": 3,

       "maxNodeCount": 10

     },

     "nodeSize": "Medium",

     "nodeSizeFamily": "MemoryOptimized",

     "sessionLevelPackagesEnabled": true,

     "sparkEventsFolder": "/events",

     "sparkVersion": "3.1"

   }

Then, update the Spark pool using the Azure CLI:

   az synapse spark pool update --name your-spark-pool-name --workspace-name your-workspace-name --resource-group your-resource-group --file @spark-pool-config.json