Alert is being suppressed for all servers even only one server is in maintenance

Question

Alert is being suppressed for all servers even only one server is in maintenance

Natalia Efimtceva 21

Hello dear Azure team,

We faced a business affecting case when critical event alert was suppressed for production servers.

Case description
There is a list of the servers, each server has it’s own maintenance time for example in OOH or for patching.
Alert Processing rules are configured to suppress Alerts generated on these servers during maintenance window by schedule and filter: Alert context (payload) – Contains - “names of the server in maintenance”.

The issue
Alert applied to the list of servers are being suppressed for all servers from the list if even one server is in maintenance.
So if similar alert is fired on two servers one of which is in maintenance but second is alive both Alerts are being suppressed.

Possible solution we tried
According to https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-processing-rules?tabs=portal filters can be used to exclude needed resources.

But even if we put server resource to exclusion, in the scenario described above alert is also suppressed.

Could you please kindly assist/advise how resolve the issue or overcome this Alert processing rules limitation?
Thanks.

Accepted answer

1 additional answer

Your answer

Answer 1

Stanislav Zhelyazkov 28,676 MVP Volunteer Moderator

Hi,
Best is to check what kind of json payload your alert will generate. If the provided value is matched somewhere in the payload it is expected. It is important that your alert rule fires per resource (computer in your case) otherwise if you have one alert that fires for all computers it is expected that will suppress it.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Natalia Efimtceva 21 Reputation points

2022-12-19T07:48:21.317+00:00

Thank you for the recomendations.
We found out that option "Split by dimentions" could help in out case (and ResourceId should be presented in the search query to be able to choose this option). Unfortunately, our Log Analytics Workspace and Alert Processing Rules are outdated (old preview API) so we have to migrate to new APIs to use "Split by dimentions" feature.
Stanislav Zhelyazkov 28,676 Reputation points MVP Volunteer Moderator

2022-12-19T12:03:33.46+00:00

Here is more information on Log Alert v2 if you are referring to it. Alert processing rules have new API but it is not so different from the previous ones in terms of how processing rules work. On the other hand Log Alert v1 and v2 have some important differences which I covered in the blog post. Please mark the reply as answer if it has helped you.
Natalia Efimtceva 21 Reputation points

2022-12-20T04:54:04.82+00:00

Stan, thank you. Your article is very useful. Microsoft engineer also confirmed that that for legacy Alert v1 the VMs are not captured in the alert payload and the alert processing rule cannot recognize which resource to suppress. In order to have control over which resources are captured in the context, we have to use the log search v2 alerts and split by dimensions.

Right now for Log Analytics Workspace we have 2015-11-01-preview. Do you know if we could upgrade it to the new api version in place (without creating new Log Analytics Workapce)? We have all alerts defined in code (ARM templates) - so could we just update API version in the tempate and apply it?
Thank you.
Stanislav Zhelyazkov 28,676 Reputation points MVP Volunteer Moderator

2022-12-20T11:32:47.657+00:00

So Log Alerts and Log Analytics workspace are two separate resources that does not have dependency on each other besides referencing the workspace resource ID when you create Log Alert. You can update the existing alerts by just using the new API version for Log Alerts. Of course it is better to modify the alerts' properties to accommodate to the differences.
Natalia Efimtceva 21 Reputation points

2022-12-20T14:29:29.897+00:00

Yes, that is correct - it's different resources and not depending on each other. Thank you. We also tried to update LWA API version via ARM - it works correctly.
Stanislav Zhelyazkov 28,676 Reputation points MVP Volunteer Moderator

2022-12-20T14:45:40.48+00:00

The Log Analytics workspace API does not bring any changes to how the workspace resource works. More on the topic of Log Alert v1 (api 2018-04-16) vs v2 (api 2021-08-01).

Answer 2

Hoekstra Jelle 501

Hi,

Have you tried the operator "starts with" vs-weu01 or "like" vs-weu01* (if it works) that might change some things around.

Hope these offer a remedy, elsewise please leave a comment.

----------

If it does help, please do accept the answer.

Natalia Efimtceva 21 Reputation points

2022-12-20T04:48:13.033+00:00

Thank you for the suggestion but it's not an option and there is no like operation (only contains). And the issue is related to another problem. Please see later comments.

Share via

Alert is being suppressed for all servers even only one server is in maintenance

1 additional answer

Your answer