This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Currently, we have downgraded a few pools to Spark 3.4 and attached the requirements.txt file. Everything is working well, and the startup time has returned to the usual 4–5 minutes. At the same time, we are running one Spark pool daily with Spark version 3.5 and requirements.txt. While it hasn’t failed yet, the startup time remains higher at around 10–12 minutes.
Is it advisable to continue with Spark 3.4 for now, until Spark 3.5 moves out of preview or becomes more stable for package management?
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Hi Nikhil Singh,
Welcome to the Microsoft Q&A Platform.
If Spark 3.4 is working well for you and meets your needs, it makes sense to stick with it especially for critical workloads until Spark 3.5 becomes more stable.