How to Delay a (Warning) Alert on the Console

Question

Hi Team,

Trying to figure out if its possible to delay a Warning Alert to be presented on the scom console.

If a Warning Threshold for a Low Disk Space has been triggered, I don't want it to go to the console straight away.
I want this to be delayed say about 45min etc just in case the Disk Space recovers in this time and the alert will not be required anymore.
And if the threshold is still in a breached state after 45min, only then fire a Warning Alert.

Is this possible using Overrides or do I need to create a New Unit Monitor to achieve this?
If a new monitor is required, what do I need to achieve this, a 3 state monitor > From Healthy to Warning to Healthy > but where and how do I set the delay part?
Is it in the "consecutive samples" - if so > what should be the Interval and Samples

Answer

Thanks XinGuo,

So can you confirm the process below that I should follow:
1 - A new Unit Monitor > Win Performance Counter > Static Threshold > Consecutive Samples over Threshold
2 - I should adjust my consecutive samples to delay the alert until the consecutive samples count is True
3 - Now since I have my new unit monitor working, I will get double Alerts being presented in the console. One from my unit monitor and the other from the Default Win Server Operation System MP that monitors Logical Disk.

So in order to rectify this, I should Override the ENABLE value to FALSE:

Is this correct so far?

Answer

Hi @Saiyad Rahim ,

what @XinGuo-MSFT posted about alert supression is not relevant in your case (I suppose). Alert supresson means that one alert is always generated and then the generation of all subsequent alerts is managed by the alert supression settings.
In your case, you want the first alert to be delayed, you don't want to have one alert and then supress all following alerts, right? This is not possible and also does not make a big sense..The goal of a monitoring tool is to inform you as soon as possible as an issue is detected.
Still, I think I understand why you have the requirements. My advice to you would be: adjust your process accordingly.
Of course, if we are talking about notifications or Incident creeation there are other ways to introduce a delay. But an alert will be shown in the console right after an issue is detected.
This being said you can do the following (this is just an example, there might be easier ways to do this)

Create a Monitor, which monitoris the disks Space, but has a lower Threshold then the current one and does not Alert.
Configure a Recovery Task (PowerShell script or similar), which waits for 45 Minutes, then calculates the disk space again and sends a mail.
Whenever you get a Mail from the recovery task script, then you know that the issue was there for 45 Min and that an alert will be generated by the default monitor.

I personally am not a fan of such approaches. I would consider some other action if I were you - either delay sending the notification for the alert or delay Incident creation...this wwould depend on the ITSM process that is in place.

Hope I would help out!

----------

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)
Regards
Stoyan Chalakov

Answer

Thanks Stoyan.

Have a follow up question -

Once I have done the above steps (created a Warning Alert and a Error/Critical Alert and Enabled Default Logical Disk Monitor to False), I assume that for a Server that has its Drive breach a Warning threshold...gets the alert on the console after 45min....and assuming that it has not been resolved automatically or by any engineer.....after a while it ends up breaching the Error/Critical threshold.

In this scenario, I assume that an Critical disk alert will be generated....so what happens to the initial Warning alert for the same drive?

I am thinking that it will still be present on the console as well until the Warning threshold issue is resolved.
Am i correct?

Does this will mean that at a particular point I can have 2 alerts for a drive active on the console, one for Warning and one for Critical?

Can not remember if this is the same behaviour of SCOMs default Disk Monitor or does SCOM close the Warning alert when the same drive goes into Critical.

Answer

Hi Saiyad (@Saiyad Rahim ),

this is your explaination, coming directly from Microsoft.. From:

How an alert is produced
https://learn.microsoft.com/en-us/system-center/scom/manage-alert-generation-overview?view=sc-om-2022

If a monitor sends an alert for warning or critical, and the monitor sent an alert when the state changed to warning, it will only send a second alert when the stage changes from warning to critical if the first alert has been closed. If the alert that was sent when the state changed to warning remains open, no alert will be sent when the state changes from warning to critical.

So you won't get Critical before the Warning is closed.
I think it would be a better aproach in your particular case if you create a 2-state monitor going directly from Healthy to Critical. This way you will get a critical only in case the threahsold is breached after those 45-60 min.

Regards,
Stoyan

Answer