How to integrate/add more metrics & info into Ganglia UI in Databricks Jobs

Question

As per https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage#monitor-performance, Ganglia metrics Collection Period Snapshot modifications can be done using init scripts.

Could you please help with pointers to modify by default Ganglia dashboard provided in Databricks jobs.

[Note: As we migrate from MSDN, this question has been posted by an Azure Cloud Engineer as a frequently asked question]

Sourced from MSDN – Azure Databricks

Answer 1

Welcome to the Microsoft Q&A platform.

Happy to answer your questions

The pre-installed Ganglia UI is not modifiable in that way. If you want to create custom dashboards, you should set up your own monitoring system by using Azure Log Analytics.

One option is to use Azure Monitor + Power BI dashboards: https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/

OR

Monitoring and Logging in Azure Databricks With Azure Log Analytics and Grafana: https://cloudarchitected.com/2019/04/monitoring-azure-databricks/

Hope this helps.

Sourced from MSDN – Azure Databricks