Azure Databricks - How to monitor AutoScaling -

Question

All,

I have enabled autoscaling on the Azure Databricks cluster. For the workload, how do I check if autoscaling kicked in? Is there a UI or a CLI to check?

Thanks,
grajee

Answer 1

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

How do I check if autoscaling kicked in? Is there a UI or a CLI to check?

Yes, you can check it directly through databricks cluster - Configuration page & Spark UI page.

Monitor databricks cluster autoscale using configuration page:

Note: When Databricks cluster is running you can view the current count of the worker type as shown below.

Monitor databricks cluster autoscale using Spark UI page:

Walkthrough on how to monitor Azure Databricks - Auto Scaling through UI:

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators