Monitor based Alerts are not auto closing in SCOM

shankar431 471

Hi Team,

We have observed Monitor based Alerts are not auto closing in SCOM 2019 environment.
When we go to the Health explorer for the particular alerts we see monitors caused the alerts are healthy.

We have onbserved this issue with multiple Monitors.

Can you please let us know how we can solve this issue.

Note: Some times when we do the health reset for the monitor Alert is closing.in that case we are closing the alerts manaully.

Regards,
Ravi shankar

3 answers

Crystal-MSFT 43,721 Reputation points Microsoft Vendor

2021-09-02T01:18:30.713+00:00

@shankar431 , From your description, it seems the alert generated by a monitor will not be resolved automatically until we reset the health state. If there's any misunderstanding, please let us know.

Here, we suggest to go to the properties of the monitor, click Alerting tab to see if the option "Automatically resolve the alert when the monitor returns to a healthy state" is selected. Here is a link with the detailed steps for the reference:
https://learn.microsoft.com/en-us/system-center/scom/manage-alert-created-by-monitor?view=sc-om-2019#to-determine-if-an-alert-is-resolved-automatically

If not, please select it and see if our issue can be fixed.

Hope it can help.

If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
Please sign in to rate this answer.
shankar431 471 Reputation points

2021-09-02T06:41:52.803+00:00

Hi Crystal,

Thank you for your response.
Yes monitor generated alerts are not getting closed after monitor turns to healthy state.

"Automatically resolve the alert when the monitor returns to a healthy state" option is already selected for the Monitors.

Regards,
Ravi shankar

Crystal-MSFT 43,721 Reputation points Microsoft Vendor

2021-09-03T02:15:16.643+00:00

@shankar431 , Thanks for the response. From your description, I know the setting is already set. But it is not working. It is a little strange. Here, we suggest to clear cache on management server and agent, then schedule a non-business time to do restart to see if it can be fixed.
https://learn.microsoft.com/en-us/system-center/scom/manage-clear-healthservice-cache?view=sc-om-2019

If there's any update, please let us know.

System Center guy 686 Reputation points

2021-09-07T04:30:03.817+00:00

Agree with crystal. More of the time, open the operations console with /clearcache option can solve your issue.

Roger

Crystal-MSFT 43,721 Reputation points Microsoft Vendor

2021-09-08T02:21:35.827+00:00

@shankar431 , Hope things are going well. I am writing to see if we have the opportunity to try our suggestions. If there's any update, feel free to let us know.

Thanks and have a nice day!

shankar431 471 Reputation points

2021-09-08T10:54:46.38+00:00

Hi Crystal,

Do i need to celar cache on all the SCOM Management servers and restart required on all ?
Also after clearing cachce on agent machines also do i need to restart.

Can we clear cache for agent installed servers and Management servers from SCOM Montiroing pane ->Task status view - clear cache or do we need to delete the Health state folder.

\Program Files\System Center Operations Manager\Agent\Health Service State

\Program Files\Microsoft System Center\Operations Manager\Server\Health Service State

Please confirm on this.

Regards,
Ravi shankar

Crystal-MSFT 43,721 Reputation points Microsoft Vendor

2021-09-09T05:43:10.023+00:00

@shankar431 , Here are answers tor your questions:
Q1:Do i need to celar cache on all the SCOM Management servers and restart required on all ?
A1: We can firstly open the Operations console with /clearcache on one server to see if the result will be different. If not, try to run Flush Health Service State and Cache on all the management servers to see if the result will be different. If still not, try to schedule s restart.

Q2:Can we clear cache for agent installed servers and Management servers from SCOM Montiroing pane ->Task status view - clear cache or do we need to delete the Health state folder.
A2: Yes, if the agent is responsive the task will do similar thing. If the agent is not responsive, we can stop the health service and remove the folder than start the service again. After that, a restart will be better.

Hope it can help.

Crystal-MSFT 43,721 Reputation points Microsoft Vendor

2021-09-15T03:10:13.397+00:00

@shankar431 ,Hope the above information can help. if there's anything unclear, feel free to let us know.

shankar431 471 Reputation points

2022-12-21T14:36:05.717+00:00

Hi Crystal,

Do we need to perform the above two steps on agent facing Management servers will be enough or need to perform on remaining management servers dedicated for network monitoring and ACS monitoring etc we have in our environment.

Regards,
Ravi Shankar

SChalakov 10,266 Reputation points MVP

2022-12-21T15:59:35.317+00:00

Hi Ravi,

do you still refer to the original issue (Alerts from Monitors are not being auto-closed)?
Can you please post some more info about the monitor itslef and also the UR level of your SCOM environment?
Is the issue intermittent or can be reproduced at any time?

Thanks and Regards,
Stoyan

shankar431 471 Reputation points

2022-12-22T13:56:18.093+00:00

Hi Stoyan,

Yes, Iam referring to the same issue.
Below is the example. Monitor is healthy but alert not auto closed.

We are on UR3 SCOM 2019.
We have observed this is happening on irregular basis, ever for different monitors.

Regards,
Ravi shankar
Sign in to comment
shankar431 471 Reputation points

2023-01-18T13:53:32.05+00:00

Hi All,

Still this issue persists in my environment any suggestions on this to solve this issue,

Ravi shankar
Please sign in to rate this answer.
SChalakov 10,266 Reputation points MVP

2023-01-19T15:20:19.3233333+00:00

Hi Ravi,

I had this kind of behavior in two cases:

when my SCOM 2019 Management Group was not update. Which UR level are you on? Pleae make sure you have installed UR4 and all relevant hotfixes.

when a Antivirus Software was scanning the cache (console, management server). In this case you need to ass exclusion as per: Configuring antivirus exclusions for agent and components

I hope I could help you with that.

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)
Regards
Stoyan Chalakov

Thierry THOMAS 0 Reputation points

2024-02-02T15:41:58.63+00:00

Hi Ravi, I've sometime the same behavior on a Rollup monitor, sometime it works fine, alert is closed immediately, and sometime it's 3 hours later, I've you find a resolution? My environment is SCOM 2019 with UR5.

Regards, Thierry
Sign in to comment
Marius Ene 335 Reputation points

2024-02-05T15:08:50.1366667+00:00
Hello,

Please ensure that the monitors in question don't have the 'manual reset' configuration.

Assuming that the monitors have an automated health check condition, you need to look for the following possible issues:

Check the SCOM agent client logs for communication issues with the SCOM server. Also errors related to agent send queue limit reached.

Check for dropped workflows in the client logs. Sometimes AV might block certain scripts and workflows causing them to be dropped altogether. Restart the Health Service on the problem agent to resume the dropped workflows.

Might be a SCOM cached console issue as others mentioned.

Check for errors on the SCOM server side, in the Operations Manager logs, particularly event ids 4506 related to data drops.

Check the MG Health dashboard and confirm there are no issues present.

Good luck!
Please sign in to rate this answer.

0 comments No comments
Sign in to comment