Change Spark Pool Version in Synapse Analytics

Question

Hi Guys, i hope you are good!
I have a Synapse Env with a lot of Spark Pools with spark version 3.4. I need to change the pools to the version 3.5, we have a path to do this easier? I search in the documents and in the AI but i just saw that i can do this manually, deleting the old cluster and create a new cluster. But this is impossible for me, i have around 1000 notebooks connected in 20 spark pools, and i found is impossible to delete a cluster if it is connected in a notebook. I'm desperateeeee :(

I can't use the PowerShell too, because is a office problem and they dont't agree. I don't have GIT too, i know that is a possibility and is the easier path

Answer 1

The Synapse Studio UI does not let you change the Spark version of an existing pool. This is by design and is officially confirmed in the Microsoft Learn

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-version-support

As you can see it is indicated like : "This isn't allowed from UX, customer can use Azure PowerShell to update Spark version. Use 'ForceApplySetting', so that any existing clusters (with old version) are decommissioned."

Your assumption that "you can't delete a pool if it's connected to a notebook" is actually not correct. In Synapse, notebooks reference a pool by name. You can delete the pool; the notebooks remain intact and will simply fail to run sessions until a pool with that name exists again.

for solution of you do not have powershell or git you can go with REST API option

for REST API no powershell module install required.

this link may help:

https://techcommunity.microsoft.com/blog/azuresynapseanalyticsblog/getting-started-with-rest-apis-for-azure-synapse-analytics---apache-spark-pool/3668474

or alternative you can use cloud shell if your policy is to block powershell locally you can use this sessions from brower s well

https://learn.microsoft.com/en-us/powershell/module/az.synapse/update-azsynapsesparkpool

Answer 2

Hi Guys, i already did this, we can do this using the Portal Azure PowerShell, we can fix it doing this:

1 - Create a new auxiliary cluster

2 - Move all the notebooks using powershell to this auxiliary cluster

3 - Change the spark version in your cluster using powershell too

4 - Move all the notebooks using powershell to the old cluster

In this case we wont need to change the access control ('Compute Operator') and we will can change the spark version.

The code in powershell is that:

$workspaceName = ""

$oldPoolName = "" #your pool

$newPoolName = "" #aux pool

$resourceGroup = ""

Write-Host "`n[2/3] refresing notebooks by '$oldPoolName' to '$newPoolName'..." -ForegroundColor Cyan

 

$tokenObj = Get-AzAccessToken -ResourceUrl "https://dev.azuresynapse.net"

$token = if ($tokenObj.Token -is [System.Security.SecureString]) {

   [System.Net.NetworkCredential]::new("", $tokenObj.Token).Password

} else {

   $tokenObj.Token

}

$headers = @{

   "Authorization" = "Bearer $token"

   "Content-Type" = "application/json"

}

$baseUrl = "https://$workspaceName.dev.azuresynapse.net"

$apiVer = "api-version=2020-12-01"

 

# List all notebooks (with pagination)

$notebooks = @()

$url = "$baseUrl/notebooks?$apiVer"

do {

   $resp = Invoke-RestMethod $url -Headers $headers

   $notebooks += $resp.value

   $url = $resp.nextLink

} while ($url)

 

Write-Host "Total notebooks founds: $($notebooks.Count)"

 

$updated = 0

$skipped = 0

 

foreach ($nb in $notebooks) {

   $nbName = $nb.name

   $nbDetail = Invoke-RestMethod "$baseUrl/notebooks/$([Uri]::EscapeDataString($nbName))?$apiVer" `

                   -Headers $headers

 

   $poolRef = $nbDetail.properties.bigDataPool.referenceName

 

   if ($poolRef -eq $oldPoolName) {

       Write-Host " → Refreshing: $nbName" -ForegroundColor Yellow

       $nbDetail.properties.bigDataPool.referenceName = $newPoolName

 

       $body = $nbDetail | ConvertTo-Json -Depth 50

       Invoke-RestMethod "$baseUrl/notebooks/$([Uri]::EscapeDataString($nbName))?$apiVer" `

           -Method PUT `

           -Headers $headers `

           -Body $body | Out-Null

 

       Write-Host " ✓ Refreshed" -ForegroundColor Green

       $updated++

   } else {

       Write-Host " — $nbName (pool: '$poolRef') — Skipping" -ForegroundColor DarkGray

       $skipped++

   }

}

 

Write-Host "`nResumo: $updated refreshed, $skipped skipped." -ForegroundColor Cyan

The code to update version:

Update-AzSynapseSparkPool -WorkspaceName '' -Name '' -SparkVersion 3.5 -ForceApplySetting