Azure Databricks | Scala on High Concurrency Cluster Mode

Question

We are using scala notebooks with high concurrency clusters.
So far what we have read is that Scala notebooks would not benefit from high concurrency clusters, is it true?

These clusters are supposed to be shared with various other teams.

?

Answer 1

Hello @Lokesh ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is that Scala notebooks would not benefit from high concurrency clusters ? please do let us know if its not accurate.
You understanding this is correct , Scala does not benefit from high concurrency clusters . This is called out here : https://learn.microsoft.com/en-us/azure/databricks/clusters/configure#--high-concurrency-clusters

The reason being that in the high concurrent mode the cluster needs to run the workload of different users , but Scala code will be executed inside the Spark JVM (per machine) that is shared between all users, so you can get access to everything that is inside JVM.

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators