Apache Spark Pool Version Upgrade: 3.3 to 3.4

Question

What are the steps to upgrade an existing Apache Spark pool from version 3.3 to 3.4?

Answer 1

Hi @Naresh Vinjam
Welcome to Microsoft Q&A platform and thanks for posting your query here.

Azure Synapse does not support direct upgrades for existing Apache Spark pools. Instead, you’ll need to create a new Spark pool with version 3.4 and migrate your workloads.

Steps to Create a New Spark Pool with Apache Spark 3.4:

1. Navigate to Synapse Studio:
   • Open your Synapse workspace and go to the Manage hub.
   1. Create a New Apache Spark Pool:
      • Under the Apache Spark pools section, click on + New.
        • In the Spark version dropdown, select Apache Spark 3.4.
          • Configure other settings (node size, autoscale, etc.) as per your requirements.
            • Click Create to provision the new Spark pool.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

@Naresh Vinjam - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer. 

Ask: What are the steps to upgrade an existing Apache Spark pool from version 3.3 to 3.4?

Solution: I just ran the below script in PowerShell, and it successfully upgraded to version 3.4

Update-AzSynapseSparkPool -WorkspaceName XXXXXX -Name XXXXX -SparkVersion 3.4 -ForceApplySetting

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information. 

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue. 

........................................................................... 

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.