Configure Azure metric alert to trigger based on time passed?

Question

I'm just wondering if it is possible to trigger an alert on a metric ONLY if the alert has firing for a specific amount of time. I understand that some metrics support dynamic criteria so you can set a violation amount, but not every metric supports this. It also seems more intuitive to say "if this alert has been firing for 10 minutes without resolving itself, then trigger the alert". This would make creating alerts for AKS resources such as pods/containers so much easier. I don't necessarily care if a pod fails only for a new, healthy pod to take its place a few minutes later. Or if the autoscaler kicks in to spin up new pods. So in summary, I'd like to create metric alerts that trigger only if the alert has been firing for x amount of time without resolution.

Answer 1

Aaron Turcott, thank you for posting this question.

Based on my understanding, you would like to get alert only when the alerting condition has remained true for a while. Please correct me if I am wrong.

If you go via the "Alert creation" scenario as mentioned in the question, where Azure triggers an alert on a metric ONLY if the alert has firing for a specific amount of time - this creates additional alerts which are not actionable. Therefore, this is not a scenario which is readily available.

Instead, you could achieve the similar outcome using the combination of "Aggregation type" and "When to evaluate" condition in alert rule condition:

The setting above translates to - check for average for metric value every 15 minutes and if it is <10 for the last 15 minutes generate alert.

Another similar outcome could also be achieved if you are forwarding the metrics to Log Analytics workspace using the "Log based alert rule". There, an option is available to trigger after only X violations of alerting condition have happened. For example, consider the following:

In this case, the query is run every 5 minutes, but an alert is raised only if there are 2 violations, i.e., the alert will be generated after 10 minutes only.

You may choose to use either of the options to achieve the end goal.

Hope this helps.

If the answer did not help, please add more context/follow-up question for it, and we will help you out. Else, if the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

Answer 2

Hi @Aaron Turcott

Thanks for reaching Microsoft Q&A

I think wahat you are looking for is this option in the link below.

https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-processing-rules?tabs=portal#what-should-this-rule-do

monitorCondition-> When an alert fires, the alert's monitor condition is set to Fired. When the underlying condition that caused the alert to fire clears, the monitor condition is set to Resolved.

Hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.