Stop Spark cluster

Question

As I'm understanding it, a notebook runs under a spark session (application) for which many sessions can run within a pool. Also, the 15 minute auto-disable on the pool comes in to effect once all sessions/applications have completed, is that right?

Is there a way to bring down/disable the pool programmatically? Rather than waiting the 15 minutes, I want to be able to disable to pool earlier (only in certain circumstances, not every time) but all I can find are commands to stop a session ... but I assume that would only initiate the 15 minute countdown

Answer 1

Hello @Ryan Abbey ,
Thanks for the question and using MS Q&A platform.

As I'm understanding it, a notebook runs under a spark session (application) for which many sessions can run within a pool. Also, the 15 minute auto-disable on the pool comes in to effect once all sessions/applications have completed, is that right?

Correct. Once all the sessions are completed, idle time kicks and the cluster is terminated thereafter.

Is there a way to bring down/disable the pool programmatically? Rather than waiting the 15 minutes, I want to be able to disable to pool earlier (only in certain circumstances, not every time) but all I can find are commands to stop a session ... but I assume that would only initiate the 15 minute countdown

I don’t think this is possible. if you do not want to wait for 15 min, then we can lower the idle time setting at the pool level.I did tried a time of 5 min and atleast I was able to deploy the cluster .

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators