Instance Pools in Azure Databricks (WHY?)

David Beavon 991 Reputation points
2021-03-27T22:39:06.88+00:00

The instance pools in azure databricks allow you to define a custom pool of VM's that are warmed up and ready for use. You can specify a min idle quantity which will ensure that there are always some on hand when you need them. The end result is that your clusters can be started several minutes faster than otherwise.

The instance pools are nice. But shouldn't this be the way things behave in the first place? I always thought that the whole point of doing computing in the cloud that we share a pool of resources with lots of other customers. That introduces efficiencies. One of them should be, for example, that a cluster of VM's are already prepared for use and don't need to be booted from scratch.

There is little that can be customized in the custom instance pool. We use standard CPU "Standard_DS3_v2", RAM, cores, and we specify the version of the LTS databricks runtime. These selections should be common to a ton of other customers. Given these common factors, it seems that databricks should be able to manage a pool on my behalf. If anything, they should just display a checkbox that says "Your cluster looks pretty typical, do you want to use our standard instance pool to improve startup times?" : YES/NO.

The main things that bother me about these custom instance pools:

  • Seems unnecessary. We are already using a pool of azure cloud resources. Why am I defining a pool within a pool?
  • Cost burden. I'm having to pay for a pool of idle VM's. Every customer should not have to pay the cost of having their own distinct set of idle VM's, especially if lots of other customers have the same idle VM images as well. It is likely that everyone is paying many times over for the same underlying set of idle VM's.
  • Configuration management. What a lot of hoopla! In the right scenarios, none of this configuration cruft shouldn't be needed (ie. when using a common DB runtime, CPU, RAM, and core selection). There should be a simple checkbox where you can choose to use the standard instance pool.

What am I missing? Is this just a new feature request that nobody has ever considered? Or is there some design principal that I'm overlooking?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,330 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.