Exchange 2010 Correlation Engine Not Generating Alerts

While helping a customer migrate from SCOM 2007 to SCOM 2012 R2, we noticed a large reduction in Alerts for Exchange 2010.  After investigating further, we noticed that we were not getting any Alerts that come from the Exchange 2010 Correlation Engine.

After verifying with my customer that the Exchange 2010 Monitors were generating state changes, but no alerts were coming in, we reviewed the event log for the Correlation Engine (it logs to the Application Event log on the server it is installed on) and saw event 717 being logged every 5 minutes (more details below).

After doing some research, I found the following information which resolved the problem (provided by Microsoft Escalation Engineer Manoj Parvathaneni).

Quick Summary:
We had to install the Correlation Engine on a server without any SCOM 2012 server components installed, and include some SCOM 2007 dlls in the Correlation Engine installation directory.

Symptoms

When the Exchange Correlation Engine is installed on a System Center 2012 Operations Manager Management Server, the Automatic Alert Resolution feature in the Exchange Correlation Engine ceases to function and the following event is logged in the Application event log on the Management Server:

Log Name: Application
Source: MSExchangeMonitoringCorrelation
Event ID: 717
Task Category: General
Level: Warning
Keywords: Classic
User: N/A
Description:
Connection with the Operations Manager Root Management Server failed.

Error: System.InvalidCastException: [A]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo] cannot be cast to [B]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo]. Type A originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'. Type B originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'.
at Microsoft.EnterpriseManagement.Utilities.GetDeserializeObject[T](Byte[] serializedObject)
at Microsoft.EnterpriseManagement.DataAbstractionLayer.InstanceSpaceOperations.UpdateAlerts(Byte[] alerts, String comments, Nullable`1 modifyingConnectorId)
at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateAlertsInternal[T](IList`1 alerts, String comments, Nullable`1 modifyingConnectorId, ManagementGroup managementGroup)
at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateInternal(String comments, Nullable`1 modifyingConnectorId)
at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.Update(String comments)
at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlert(Entity entity, MonitoringAlert monAlert)
at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlerts(Entity entity, String alertCriteriaString)
at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateWithinEntity(Node entity)
at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateBatchTask(Object batchData)

Number of occurrence: 3

You'll also notice the following error in the Exchange Correlation Engine logs:

2013-11-18T08:16:15.673Z,4,Information,"CorrelateBatchTask: caught exception [System.InvalidCastException: [A]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo] cannot be cast to [B]System.Collections.Generic.List`1[Microsoft.EnterpriseManagement.Monitoring.MonitoringAlertUpdateStatusIndigo]. Type A originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'. Type B originates from 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'. at Microsoft.EnterpriseManagement.Utilities.GetDeserializeObject[T](Byte[] serializedObject) at Microsoft.EnterpriseManagement.DataAbstractionLayer.InstanceSpaceOperations.UpdateAlerts(Byte[] alerts, String comments, Nullable`1 modifyingConnectorId) at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateAlertsInternal[T](IList`1 alerts, String comments, Nullable`1 modifyingConnectorId, ManagementGroup managementGroup) at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.UpdateInternal(String comments, Nullable`1 modifyingConnectorId) at Microsoft.EnterpriseManagement.Monitoring.MonitoringAlert.Update(String comments) at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlert(Entity entity, MonitoringAlert monAlert) at Microsoft.Exchange.Monitoring.CorrelationEngine.MomSdkProxy.ResolveEntityAlerts(Entity entity, String alertCriteriaString) at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateWithinEntity(Node entity) at Microsoft.Exchange.Monitoring.CorrelationEngine.CorrelationEngine.CorrelateBatchTask(Object batchData)].

In addition, the Exchange Correlation Engine will not report any new Alerts or report the alerts to SCOM 2012 inconsistently.

Resolution

The exception above can be resolved by using one of the following methods:

Method 1: Install Exchange Correlation Engine on a dedicated system without the SCOM 2012 Management Server component

  1. Install Exchange Correlation engine on this machine.
  2. Stop the Microsoft Exchange Monitoring Correlation service.
  3. Copy SCOM 2007 R2 SDK binaries into the CE install directory (Program Files\Microsoft\Exchange Server\v14\Bin).
    • Microsoft.EnterpriseManagement.OperationsManager.Common.dll
    • Microsoft.EnterpriseManagement.OperationsManager.dll
    • MomBidLdr.dll

**UPDATE: Files attached to this blog

Where can you find the above binary files?

  • The SCOM 2007 R2 SDK Binaries Microsoft.EnterpriseManagement.OperationsManager.Common.dll and Microsoft.EnterpriseManagement.OperationsManager.dll can be found on a system that has SCOM 2007 R2 Console in the following directory Program Files\System Center Operations Manager 2007\SDK Binaries.
  • The MOmBidLdr.dll can be found can be found on a system that has SCOM 2007 R2 Console in the following directory Program Files\System Center Operations Manager 2007

Method 2: Disable the Automatic Alert Resolution Feature in the Correlation Engine

By following this approach the customer will lose the ability to resolve the alerts automatically via the Exchange Correlation Engine.

To disable Automatic Alert Resolution, perform the following steps:

  1. Log on to the server that's hosting the Microsoft Exchange Monitoring Correlation Engine service.
  2. Locate the Correlation Engine configuration file named Microsoft.Exchange.Monitoring.CorrelationEngine.exe.config. By default, the file is located in C:\Program Files\Microsoft\Exchange Server\V14\Bin\, where C:\ is the Exchange installation directory.
  3. Open Microsoft.Exchange.Monitoring.CorrelationEngine.exe.config in a text editor such as Notepad
  4. Locate the following line in the configuration file:
    <add key="AutoResolveAlerts" value="true" />
  5. Change <add key="AutoResolveAlerts" value="true" /> to <add key="AutoResolveAlerts" value="false" />
  6. Restart the Microsoft Exchange Monitoring Correlation Engine service.

More Information

Exchange Correlation Engine:
The Correlation Engine(CE) is a stand-alone Windows service that uses the Operations Manager SDK interface to first retrieve the health model (or instance space) and then process state change events. By maintaining the health model in memory, and processing state change events, the Correlation Engine is able to determine when to raise an alert based on the state of the system. The Correlation Engine solely relies on the Operations Manager SDK. It does not use the agent to perform any of the correlation and other aspects of its operations. For further details, you can also reference the Exchange 2010 Management Pack guide.

Automatic Alert Resolution feature in Exchange Correlation Engine:
The Automatic Alert Resolution feature automatically closes related alerts when the Exchange 2010 Management Pack determines that the underlying issue is no longer a problem. This feature is provided by the Correlation Engine, and is enabled by default. Using Automatic Alert Resolution can cause multiple alerts to be logged if the same alert is logged again for another instance of the problem before the associated ticket has been resolved by support teams.

You may also want to disable this feature under the following or other conditions:

  • If you're using ticketing or another support system that wouldn't work correctly if alerts are automatically resolved.

If you're using a connector with Operations Manager 2007. A connector is a custom service or program that allows Operations Manager to communicate with external systems. For example, you may want to disable this feature if you're using a connector that allows an external application to track Exchange 2010 Management Pack alerts.

SCOM2007FilesForEXCHCorrelationEngine.zip