Potential issue on Alerts management process

Many time Operators team manages problem regarding OpsMgr2007 Alerts views.

When an alert is raised, an operator acknowledges the alert, manages the issue, and closes the alert if not already done.

The following chapters explain why this can be an issue, and how to manage it.

Alert concept in OpsMgr2007

2 types of alert are managed in OpsMgr2007.

Ø  Alert generated by Rule

Ø  Alert generated by Monitor

Alerts generated by Rule

This type of alert is generated by an OpsMgr2007 rule which doesn’t affect the health of the target object (Alert Source).

In alert details windows, a link appears on the Alert Rule:

Alert generated by rule can be configured to consolidate if the problem is raised again. In this case the Repeat Count field is updated:

Alert context tab contains the last event which has been used to generate the alert:

 Auto resolved alert process affects only alert generated by rule, if alert is still in state NEW:

Generally rules are used to collect information, as events or performance counters. This information is used for troubleshooting, analysis, capacity planning, reporting …

Rules are also sometime used for proactive monitoring, in this case the rule is configured to generate alert.

Converted management packs can also content rules which generate reactive alerts as MOM2005 didn’t have monitor concept.

This type of alert is auto resolved only by the OpsMgr2007 auto resolved process, if this one is still in resolution state “New”, or if the alert source is healthy.

Alert generated by Monitor

This type of alert is generated by an OpsMgr2007 Monitor which affects the health of the target object (Alert Source).

In alert details windows, a link appears on the Alert Monitor:

Repeat count field is never updated.

Alert is used as a notification when the monitor updates the heath state of an object to Healthy to Critical (or Warning).

The alert context tab contains the event which has been used to change the monitor health state and generate the alert:

How to manage this behavior

As it is not possible to prevent an alert that has been generated by a monitor being closed by an operator and therefore not possible to ensure that the health state of the monitor has also been reset to healthy,  I have developed two tools to manage this behavior:

Ø  ResetMonitorFromAllClosedAlerts.exe

Ø  ResetMonitorfromAlertId.exe

 

 

ResetMonitorFromAllClosedAlerts

This tool scans all closed monitor alerts and checks the state of the related monitor, and if the monitor is not healthy, the state is reset.  At the next occurrence of the monitor after this has run, if the issue in question is still occurring then a new alert will be raised.

Also, to be sure that the scanned alert is the alert related to the last time the monitor state has changed, the tool will compare the time the alert was added and the last health state change value of the monitor.  This needs to be less than 90 seconds, which is a reasonable indicator that this alert and health state change are related.

This tool can be launched from the command line on the RMS server.

Without option, the tool doesn’t reset any monitor, but shows all monitors that should be reset.

With option –r, all detected monitor will be reset.

 ResetMonitorFromAllClosedAlerts.exe –r

ResetMonitorfromAlertId

Using the same principle, this tool takes in argument of an alert ID, and if it is an alert raised by a monitor which has been subsequently closed, it checks the state of the related monitor, and if the monitor is not healthy, the state is reset.

This tool can be launched automatically by creating a notification channel.

The detail of this implementation is explained in chapters below.

How to implement “ResetMonitorfromAlertId” tool.

OpsMgr2007 SP1 - Create a notification to launch “ResetMonitorfromAlertId” tool when alert is closed.

Create a new Command notification Channel

Create a new subscriber

Create a new subscription

OpsMgr2007 R2 - Create a notification to launch “ResetMonitorfromAlertId” tool when alert is closed.

Create a new Command notification Channel

Create a new subscriber

Create a new subscription

Monitoring

Events created

A rule can be created to collect the following event from RMS.

Maximum number of asynchronous responses configuration on RMS Server

As it’s described in the following blog article the hardcoded limit is maximum 5 asynchronous responses in OpsMgr2007 SP1.

https://blogs.technet.com/cliveeastwood/archive/2008/04/16/some-more-command-notification-tricks-and-tips.aspx

So if more than 5 alerts are closed in the same time, the following event should appear in Operation Manager Event log on the RMS server.

Alerts “Script or Executable was Dropped”

“The process could not be created because the maximum number of asynchronous responses (5) has been reached, and it will be dropped. Command executed: ………”

This limit can be modified by changing the following registry key:

“HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Modules\Global\Command Executer”

·         Create Keys: Global\Command Executer

·         Create a DWORD value called “AsyncProcessLimit" and set it between 1 and 100.

Outside of this key, it will default back to 5

ResetMonitor_Tools.zip