Is there a way to check the over-all compute usage of Databricks sql warehouse ?

Question

I am only able to see the query profile and spark UI of a particular query. Let say, if there are 14 queries running on single x-large sql warehouse, is it possible for me to check the overall compute usage of the sql warehouse ?

Answer 1

@veerabhadra reddy kovvuri - Thanks for the question and using MS Q&A platform.

Here is the response from the internal team: Unfortuantely, there is no way to check the overall compute usage of databricks sql warehouses.

Here are the important points to be noted:

• When working with SQL warehouse things are made simple by design, so the details which you are looking for are not available out-of-the-box.
• In general, you have autoscale that helps with horizontal scaling in other words concurrency, but user needs to configure the cluster vertical size (which is still a bit of challenge).  A query when run on SQL warehouse will either fail, get queued or run successfully. From an end user perspective, you want to  investigate the query rather than monitor the overall cluster (unlike an all-purpose cluster).
• Serverless overcomes the above challenge and automatically scales horizontally and vertically, hence the need to monitor the cluster may not be required, in such setup.

As per this document, we can monitor query and cluster details and could not information about total number of nodes, CPU usage, and memory usage.

Total number of nodes: cluster count X T-Shirt size of cluster
Total CPU usage: not available (could not find it)
Total memory usage: partially available (can be found for a specific query, but not overall usage)

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.