Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
508 questions
How to monitor the materialize cache size
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJY%3C/text%3E%3C/svg%3E)
Jianbo Yao (Centific Technologies Inc)
20
Reputation points Microsoft Vendor
Kusto function materialize has a limitation of 5GB size. This info was introduced in this article :https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/materializefunction
Materialize has a cache size limit of 5 GB. This limit is per cluster node and is mutual for all queries running concurrently. If a query uses materialize() and the cache can't hold any more data, the query will abort with an error.
If the amount of concurrency is large, does it mean that it is more error-prone? It will abort with an error if the cache can't hold more data. I want to know how to monitor how many spaces I used of materialize cache? Please help on this question. Thanks.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,261 Reputation points2023-10-20T18:09:13.0833333+00:00 Hello @Jianbo Yao (Centific Technologies Inc)
Yes, if the amount of concurrency is large and many queries are using the materialize() function, it can increase the likelihood of hitting the cache size limit and causing queries to abort with an error.
You can try the .show queries command to monitor the queries that are currently running and their resource usage.
-
Jianbo Yao (Centific Technologies Inc) 20 Reputation points Microsoft Vendor
2023-10-22T22:45:02.5433333+00:00 How do I know that materialize() usage has reached the 5GB limit? Is there any performance counter we can used to monitor？
-
Jianbo Yao (Centific Technologies Inc) 20 Reputation points Microsoft Vendor
2023-10-23T02:51:45.81+00:00 I defined a tabular variable. And this variable only uses once in my kusto query. According to my understanding, since I only use once, so I don't need function materialize(), please note, I didn't use materialize view, I only use materialize(). If I remove materialize() the data scan is 1.2TB. When I add materialize() back, the data scan is 30G. I don't know why? please help, thanks.
-
QuantumCache 20,261 Reputation points
2023-10-24T20:06:37.78+00:00 @Jianbo Yao (Centific Technologies Inc)
If you find that materialize() is not necessary for your query and is causing issues with cache size limits, you can try removing it and see if the query still performs well?
-
QuantumCache 20,261 Reputation points
2023-10-31T05:46:29.22+00:00 Hello @@Jianbo Yao (Centific Technologies Inc) Just checking if we are still connected on this discussion?
Did you find any resolution to the issue? Please let us know!
Sign in to comment