Summary rules - Limit on total aggregated size

Question

Folks,

I'm trying to use summary rules to aggregate firewall logs. There's a hard size limit from MS per result of 100 MB which I think is not up to the mark for firewall logs. While summarizing I'm creating two sets and grouping by 7 other fields (I need for alarms). The summary rule works 60% of the time, 40% of the time it fails because of a spike in logs that leads to the size exceeding 100 MB. I can't think of a way to workaround this. Anybody figured it out yet?

Probably can create a limit in kql to check for size and ignore anything above 100 MB but I can't think of a way I can do this in kql.

Answer 1

Hello Khanna, Keshav,

Here are some references for summary rules(preview) in Sentinel:

Aggregate Microsoft Sentinel data with summary rules | Microsoft Learn

Aggregate data in a Log Analytics workspace by using summary rules (Preview) - Azure Monitor | Microsoft Learn

The limit for summary rules is 100MB as follows,

Azure Monitor service limits - Azure Monitor | Microsoft Learn

There isn't a possible way to increase this but to avoid failures, you can try limiting the bin size or use count parameter and limit the output count. There is no direct way of filtering based on output size in KQL. 

If you find the information above helpful, please Click Yes. This will assist others in the community who encounter a similar issue, enabling them to quickly find the solution and benefit from the guidance provided.

Answer 2

Hello @Khanna, Keshav,

Thank you for posting your query on Microsoft Q&A.

Yes, you are correct—there is a limitation on summary rules where the maximum result set size is 100MB.

To address this limitation, you might consider adding a time filter to your query. The time range applied in the query will be the intersection of the filter and the bin size. You can refer to the following documentation for more details: Create or update a summary rule

Instead of processing all logs at once, partition your data based on smaller time intervals. For instance, if you're currently summarizing logs for an entire day, try breaking it down into shorter intervals, such as an hour or 30 minutes. This approach will reduce the amount of data processed in each run and help you stay within the size limit.

Please test this approach and let me know the results.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Thanks,
Raja Pothuraju.