Share via

How to confirm all spark runtime has been upgraded and meet the new requirements before end of March?

Christine Han 100 Reputation points Microsoft Employee
2026-03-03T22:32:07.1866667+00:00

Regarding Azure Synapse Runtime for Apache Spark 3.4 End of Support https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-34-runtime**
**
We have been completed the Spark pool upgrade. As we own around 50 Synapse workspaces, we want to ensure that all of them have been successfully upgraded and meet the new requirements. Could you please help verify the upgrade status under our service tree?  Service tree ids 

    Data Foundation: a3ce3b27-c434-46c2-9309-1547b569cdce  

    Datacenter Central Ticketing: 46223a8b-d1fe-4d33-8fb9-0da5f341898a

Also, could you confirm is this action is being tracked in S360 or not? We have not received any S360 items related to this yet.  Thanks for supporting.

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.


Answer accepted by question author
  1. Pilladi Padma Sai Manisha 7,055 Reputation points Microsoft External Staff Moderator
    2026-03-04T00:37:28.9266667+00:00

    Hi Christine Han,
    Yes, your overall approach is correct. Using Azure PowerShell to centrally inventory Spark pool versions across all Azure Synapse workspaces is the right method to confirm compliance before the Spark 3.4 end-of-support deadline (March 31, 2026).

    However, to make this fully accurate and enterprise-ready, a few important refinements are required.

    First, if your 50 workspaces span multiple subscriptions, the script must iterate across all subscriptions. Running it in a single subscription context may miss non-compliant Spark pools.

    Second, version comparison should not rely on string comparison (for example, "3.10" would incorrectly evaluate as less than "3.5"). Versions should be cast to [version] for reliable comparison.

    Below is the finalized, production-ready script:

    Connect-AzAccount
    
    $results = @()
    
    $subscriptions = Get-AzSubscription
    foreach ($sub in $subscriptions) {
        Set-AzContext -SubscriptionId $sub.Id
    
        $workspaces = Get-AzSynapseWorkspace
        foreach ($ws in $workspaces) {
            $pools = Get-AzSynapseSparkPool -WorkspaceName $ws.Name -ResourceGroupName $ws.ResourceGroupName
    
            foreach ($pool in $pools) {
                $isCompliant = ([version]$pool.SparkVersion -ge [version]"3.5")
    
                $results += [PSCustomObject]@{
                    Subscription     = $sub.Name
                    Workspace        = $ws.Name
                    ResourceGroup    = $ws.ResourceGroupName
                    SparkPool        = $pool.Name
                    SparkVersion     = $pool.SparkVersion
                    ComplianceStatus = if ($isCompliant) { "Compliant" } else { "Needs Upgrade" }
                }
            }
        }
    }
    
    $results | Export-Csv -Path "SparkStatus.csv" -NoTypeInformation
    

    This ensures: All subscriptions are evaluated Accurate version comparison Centralized CSV output for audit and governance review

    Regarding requirements, Spark 3.4 reaches end of support on March 31, 2026. Upgrading to Spark 3.5 GA (or later supported runtime) ensures continued security patches, bug fixes, and platform support. After upgrading, validate custom libraries, JAR/Wheel dependencies, Delta Lake compatibility, and any notebook or job runtime references.

    Regarding Service Tree and S360 tracking: Azure Support does not have direct visibility into internal Service Tree IDs for runtime validation. Compliance must be verified at the Azure resource level. Spark runtime lifecycle upgrades are platform announcements and are not automatically tracked as S360 remediation items unless your organization has integrated Azure resource compliance signals into S360 governance workflows. If you have not received S360 items, this is expected behav

    Your validation strategy is correct. With the refinements above, it will provide a complete tenant-wide compliance confirmation before the March deadline.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.