OIS 6.3 - Self-Monitoring & Event Notifications
Readers/Viewers, this is just a simple post which will be directly referenced content by the OIS 6.3 - Advanced Workflow & Best Practice Video Tutorial post.
BACKGROUND
Opalis Integration Server (OIS) has the built-in ability to “Self-Monitor”. Each object has the ability to throw an exception if certain criteria is met. By default, these event notifications are sent to the Events tab in OIS Client. In addition, these event notifications can also be sent to a SNMP Trap destination. The Opalis Event Delivery Configuration utility (oedc.exe) enables the configuration of the SNMP Trap destination(s). Once configured, each time an object throws an exception, the notification will be logged in the Events Tab (within the Client) as well as sent to the specified SNMP Trap destination(s).
In addition to the notifications generated by the Self-Monitoring functionality, events are also generated by Integration Packs when they fail to connect to third-party systems.
This post will illustrate the configuration and usage of the OIS Self-Monitoring functionality.
CONFIGURE THE SNMP TRAP DESTINATION(S)
While the SNMP Traps generated by OIS can be sent to any number of 3rd Party Network Event Management Systems (e.g. System Center Operations Manager), OIS can actually monitor the Traps its sends itself, within a separate policy, which can generate reports, take remediation actions, send notifications, etc..
| NOTE: The example within this post will show how to configure OIS so that it will Self-Monitor within its same instance. The instructions here do not require the Microsoft SNMP Trap Service to be installed or running. The Opalis Event Delivery Configuration (OEDC) can and will create a Trap Destination independent of the SNMP Trap Service. |
1. Open a Windows Command Shell.
2. cd %PROGRAMFILES(x86)%\Opalis Software\Opalis Integration Server\Management Service\
3. Type the following command:
oedc /snmp /add /ip <IP_ADDRESS> /port <PORT> /version SNMP1 /community opalis Configuration Switches:
• /snmp – the operation that instructs the utility to work with SNMP Traps
• /add – the parameter that instructs the utility to add an entry to the SNMP Traps list
• /ip – the IP address of the destination computer for the SNMP Traps (associated Action Servers)
• /port – the port used to send the SNMP Trap (Default = 162)
• /version – the version of the SNMP protocol used when sending the SNMP Trap
• /community – the community string that will be used for authenticating the SNMP Trap
HINT: The default port for the Microsoft SNMP Trap Service is 162. To use OEDC to configure the SNMP Trap Destination, ensure that the Microsoft SNMP Trap Service is not running. If it is, either utilize its functionality (instead of the OEDC) or configure the above command to utilize a different port. |
4. Execute the Command.
5. Restart the associated OpalisActionService Windows Service.
6. Repeat steps 3 - 5 for each new SNMP Trap Destination needed
Related TechNet Library Article about Event Notifications: https://technet.microsoft.com/en-us/library/gg440663.aspx
CONFIGURE OIS TO SELF-MONITOR | STEP 1 | MONITORING
Once configured through OEDC, the SNMP Trap Destination(s) can be monitored by an OIS Policy. The primary object required for this functionality is the Monitor SNMP Trap foundation object. It should be configured as follows:
Advanced Tab
NOTE: On the Advanced Tab, 1-3 Variable Bindings should be configured: • Opalis Event Type – 1.3.6.1.4.1.4217.100.100.1 • Opalis Event Summary – 1.3.6.1.4.1.4217.100.100.2 • Opalis Event Details – 1.3.6.1.4.1.4217.100.100.3 All three are shown above as Output - 1, Output -2, and Output -3. The content of SNMP traps will be truncated at 1000 characters. |
Details Tab
If the SNMP Trap Destination(s) were configured using the OEDC utility, then the No Dependency option should be used with the specified Port. Also, ensure that the IP Address specified with the OEDC utility is entered in the Source host field.
NOTE: On the Details Tab, the Connection/Trap Filter should be configured: • Connection – No Dependency • Port – <PORT> • Source Host – <IP_ADDRESS> • Enterprise Identifier – 1.3.6.1.4.1.4217.100.100 • Generic Identifier – enterpriseSpecific(6) • Specific Identifier – 1 The above was configured as a No Dependency Connection. When utilizing the Microsoft SNMP Trap Service (instead of OEDC configuration), be sure to select the Microsoft SNMP Trap Service Connection option instead (and configure the Trap Filter accordingly). HINT: Some external SNMP monitoring software may require the use of the OIS MIB file. This file (OIS.mib) is located in the ..\Support\SNMP\ folder on the OIS installation media. |
The following table lists the messages that are generated by Opalis Events:
Message |
Type |
Criteria for sending |
Description |
Action Server <ActionServerName> is reaching its maximum capacity |
Information |
An Action Server is running at its maximum capacity: the number of Policies running by this Action Server is approaching the Policy Throttling limit. |
The number of Policies running on the Action Server is approaching the maximum number allowed for this Action Server (MaxRunningPolicies value). |
An Action Server has stopped running |
Warning |
An Action Server goes down unexpectedly (as opposed to explicitly shutting down the service.) |
The Action Server <ActionServerName> is no longer running. Verify the status of the Action Server and try to restart it. If the problem persists, contact Technical Support. |
Action Server <ActionServerName> is experiencing frequent errors while accessing the database |
Warning |
Action Server experiences frequent errors while accessing the DB |
The Action Server is experiencing frequent errors while accessing the database. Verify the status of the database server. If the problem persists, contact Technical Support. |
Action Server <ActionServerName> cannot connect to the database |
Error |
Action Server cannot connect to the DB. |
The Action Server cannot connect to the database server. Verify the status of the database server. If the problem persists, contact Technical Support. |
The Policy "<PolicyName>" failed |
Warning |
If executing a Policy fails and is configured to notify on failure. |
The user requested that this Opalis Event be generated if the Policy fails. |
The object "<ObjectName>" in the Policy "<PolicyName>" failed |
Warning |
If executing a Task fails and is configured to notify on failure. |
The user requested that this Opalis Event be generated if the object fails. |
It is taking longer than expected to execute the Policy "<PolicyName>". |
Warning |
If executing a Policy takes too long |
The user requested that this Opalis Event be generated if it took the Policy longer than n seconds to finish running. The Policy has run for longer than this time. |
It is taking longer than expected to execute the object "<ObjectName>"in the Policy "<PolicyName>". |
Warning |
If executing a Policy takes too long |
The user requested that this Opalis Event be generated if it took the object longer than n seconds to finish running. The object has run for longer than this time. |
Deployed Policy is not running |
Warning |
If a Policy is published but has not been started for long time |
The Policy "<PolicyName>" is started, but is not running. Verify the status of the Action Servers assigned to run this Policy. If the Action Servers are not running, restart them. If the Action Servers are busy and cannot process more Policies, deploy more Action Servers. If the problem persists, contact Technical Support. |
Could not start the policy |
Warning |
An Action Server tried to start a published Policy, but the definition of this Policy is empty or contains only disabled objects |
The Policy '<PolicyName>' does not contain any objects. |
License warning |
Warning |
Licensing issues |
The license for one or more of the Integration Packs installed is missing or invalid. Use the License Manager utility to configure your licenses. |
License warning |
Warning |
The workload created by a Policy exceeds the current system throughput |
Not enough resources to continue running the "<PolicyName>" policy. The policy is suspended. It is not stopped and the events triggered by monitors are not lost. The execution of this policy will resume as soon as there is enough resources available |
Policy suspended: not enough resources |
Warning |
The number of threads required to start a Policy exceeds the current limit |
Policy "<PolicyName>" stopped: not enough threads to run policy |
Policy stopped: not enough threads |
Error |
Policy scheduling |
Policy "<PolicyName>" stopped: policy schedule does not permit the policy to run at this time |
Policy stopped: denied by schedule |
Warning |
Policy scheduling |
Policy "<PolicyName>" suspended: policy schedule does not permit the policy to run at this time |
Policy suspended: denied by schedule |
Warning |
Cannot resolve a computer group |
Policy "<PolicyName>" stopped: Expansion of computer group failed. |
Policy stopped |
Warning |
Cannot resolve a variable |
Policy "<PolicyName>" stopped: Expansion of Variables failed. |
Policy stopped |
Warning |
Authentication issues |
The text describing the details of the last error is retrieved from the system |
Impersonation failed |
Warning |
Authentication issues |
The text describing the details of the last error is retrieved from the system |
LogonUser failed |
Warning |
Monitor Object fails to connect to 3rd party system |
<Set by object> |
<Set by Object> |
<Custom> |
Custom Opalis Event |
<Custom> |
Related TechNet Library Article about Self-Monitoring: https://technet.microsoft.com/en-us/library/gg440746.aspx
CONFIGURE OIS TO SELF-MONITOR | STEP 2 | EVENT SOURCES
With the Monitor SNMP Trap configured and running, all new Opalis Events sent as SNMP Traps will trigger this object. Before any objects within OIS will send Opalis Events as SNMP Traps, a simple, object-level configuration needs to be set for each object requiring event handling. Once configured, these objects can be considered Event Sources.
All OIS objects have an Run Behavior tab on the Object Properties screen. On this tab, there are two options: “Report when the object runs for more than: N seconds” and/or “Report if the object fails to run”.
Run Behavior Tab
Each of these options, if configured, will send an SNMP Trap to all the specified SNMP Trap Destinations (configured with OEDC), when either of the conditions are met.
HANDLING THE SNMP TRAP DATA WITHING OIS
With the Monitor SNMP Trap object configured (as well as objects configured as Event Sources), there are many methods available to handle all the data that will be generated each time the monitor triggers based on an event. The method used is specific to the intended use case. Two methods will be illustrated within this post: Monitor SNMP Trap to Text Log and Monitor SNMP Trap to MSSQL Database.
HINT: Regardless of which method is chosen to gather the information from the Self-Monitoring Event based SNMP Traps, once collected, the data is usually reported on and some kind of Notification is sent to an appropriate group. Reporting and Notification are not covered within this post, but OIS offers many methods for each. |
Monitor SNMP Trap to Text Log
Linking an Append Line object to the Monitor SNMP Trap object is the easiest and fastest way to capture the Event Data based on the Object Events sent as SNMP Traps.
Append Line Object (Configured to Capture Event Data)
The above object, configured as is, will produce Captured Event Data as:
… 6/16/2011 5:18:45 PM :: SNMP Warning :: Summary: The object "Run Program" in the Policy "1.1 Program Execution Test" failed :: Details: The user requested that this Opalis Event be generated if the object fails. 6/16/2011 5:18:48 PM :: SNMP Warning :: Summary: The object "Invoke Web Services" in the Policy "1.0 WS Connection Test" failed :: Details: The user requested that this Opalis Event be generated if the object fails. 6/16/2011 5:19:00 PM :: SNMP Warning :: Summary: The object "Run Program" in the Policy "1.1 Program Execution Test" failed :: Details: The user requested that this Opalis Event be generated if the object fails. 6/16/2011 5:19:02 PM :: SNMP Warning :: Summary: The object "Invoke Web Services" in the Policy "1.0 WS Connection Test" failed :: Details: The user requested that this Opalis Event be generated if the object fails. … |
Monitor SNMP Trap to Database
Linking an Query Database object to the Monitor SNMP Trap object is the most robust way to capture the Event Data based on the Object Events sent as SNMP Traps.
Before the Query Database object can be used to capture the Object Event data, a table will need to be created to store the data. The following is an example DROP/CREATE MSSQL Script for such a table:
if exists (select * from dbo.sysobjects where id = object_id(N'[OIS_Self_Monitoring]') and OBJECTPROPERTY(id, N'IsUserTable') = 1) drop table [OIS_Self_Monitoring] CREATE TABLE [OIS_Self_Monitoring] ( [EventID] [int] IDENTITY (1, 1) NOT NULL , [EventDateTime] [datetime] NULL , [EventType] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [EventSummary] [varchar] (200) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [EventDetails] [varchar] (2000) COLLATE SQL_Latin1_General_CP1_CI_AS NULL |
Query Database Object (Configured to Capture Event Data)
INSERT INTO [OIS_Self_Monitoring] ( [EventDateTime], [EventType], [EventSummary], [EventDetails] ) VALUES ( ' {Object end time from "Monitor SNMP Trap"} ', 'SNMP {Output - 1 - Value from "Monitor SNMP Trap"} ', ' {Output - 2 - Value from "Monitor SNMP Trap"} ', ' {Output - 3 - Value from "Monitor SNMP Trap"} ' ) |
The above object, configured as is, will produce Captured Event Data as:
![clip_image002[8]](https://msdntnarchive.z22.web.core.windows.net/media/TNBlogsFS/prod.evol.blogs.technet.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/00/83/65/metablogapi/7776.clip_image0028_01DE5A21.jpg "clip_image002[8]")
OTHER FUNCTIONALITY TO CONSIDER
Removing all SNMP Trap Destination(s)
It may become necessary to remove the previously configured SNMP Trap Destination(s). If this is the case, use the following instructions:
NOTE: This process will remove all SNMP trap destinations previously configured. SNMP Trap Destinations cannot be removed individually. After removal of the SNMP Trap Destinations, Event Messages will only be sent to the Events Tab in the OIS Client. |
1. Open a Windows Command Shell.
2. cd %PROGRAMFILES(x86)%\Opalis Software\Opalis Integration Server\Management Service\
3. Type the following command:
oedc /snmp /clear
Configuration Switches:
• /snmp – the operation that instructs the utility to work with SNMP Traps
• /clear – the parameter that instructs the operation to clear all SNMP trap destinations
4. Execute the Command.
5. Restart the associated OpalisActionService Windows Service(s).
Configuring the Delivery Method
It may be necessary to alter the delivery method for Events Captured within OIS. There are two Opalis Event delivery methods:
• EW: the Event window in the Client
• SNMP: SNMP traps
By default, all events are sent to both the Events window (via the EVENTS table) as well as SNMP; however, also by default there are no SNMP Destinations defined. Once one or more SNMP Destinations are defined, all events are sent to those SNMP Destinations as well as the Events window. As seen within this post, alternate Destinations (SNMP Trap Destinations) can be added so that OIS can either be set up as a Self-Monitor, or a 3rd Party Network Event Management System can perform the monitoring.
The actual content of the messages sent to all Destinations (Event Window or SNMP) can also be configured (filtered). The filters can be applied to either one or both of the Event Types. To filter events, a rule must be created. Configuration consists of an ordered list of rules:
Regular expression |
List of delivery methods |
For each Opalis Event, the rules from the list created are applied in the specified order. As soon as the Summary of the Opalis Event matches the regular expression, the specified delivery methods are applied, and other rules are not processed.
For example, if the list of rules looks like this:
Regular expression |
List of delivery methods |
.*interesting.* |
EW |
.*important.* |
SNMP |
.* |
EW, SNMP |
The following Opalis Events will be delivered accordingly:
Regular expression |
List of delivery methods |
Something interesting happened |
EW |
Something really important happened |
SNMP |
Something strange happened |
EW, SNMP |
To Delete all Previously Configured Delivery Method Rules:
1. Open a Windows Command Shell.
2. cd %PROGRAMFILES(x86)% \Opalis Software\Opalis Integration Server\Management Service\
3. Type the following command:
oedc /delivery /clear
Configuration Switches:
• /delivery – the operation that instructs the utility to work with Delivery Methods
• /clear – the parameter that instructs the operation to clear all Delivery Method Rules
4. Execute the Command.
5. Restart the associated OpalisActionService Windows Service(s).
NOTE: After executing this command, the list of rules becomes empty, which is interpreted as if the list contained only one rule:
By default, all Opalis Events are sent to the Events tab in the Client. If an SNMP Trap Destination exists, all events will be sent to the SNMP Trap Destination as well. |
To Add a Rule to the List of Delivery Method Rules:
1. Open a Windows Command Shell.
2. cd C:\Program Files\Opalis Software\Opalis Integration Server\Management Service\
3. Type the following command:
oedc /delivery /add /pattern <REGEX_PATTERN> [/to ew] [/to snmp] [/to none] [/to all]
Configuration Switches:
• /delivery – the operation that instructs the utility to work with Delivery Methods
• / add – the parameter that instructs the operation to add a Delivery Method Rule
• / pattern – the RegEx Pattern to be used for the Delivery Method Rule
• / to ew – the parameter that instructs the operation to add the Delivery Method Rule to the Event Window only
• / to snmp – the parameter that instructs the operation to add the Delivery Method Rule to SNMP Trap Destinations only
• / to none – the parameter that instructs the operation to add the Delivery Method Rule to neither the Event Window nor SNMP Trap Destinations
• / to all – the parameter that instructs the operation to add the Delivery Method Rule to both the Event Window and SNMP Trap Destinations
4. Execute the Command.
5. Repeat steps 3 - 4 for each new Delivery Method Rule needed.
6. Restart the associated OpalisActionService Windows Service(s).
BEWARE THE SELF-MONITORING LOGICAL LOOP
Because OIS Self-Monitoring involves monitoring events originating from OIS itself, a circular reference is created which could possibly result in an infinite loop of event triggering-monitoring-triggering-etc. While the technology can handle the loop, its perpetuation defies logic and should avoided.
Example Scenario 1:
Company A is using OIS as their default SNMP Monitor for all SNMP Traps sent within the Infrastructure. If OIS is set up with Self-Monitoring, it too sends out SNMP Traps when it encounters internal issues warranting an event. One of the potential internal issues OIS could have is with the objects actually doing the SNMP Monitoring. If this is the case, three main problems occur: 1. SNMP Monitoring for all applications is in jeopardy, this means missing events, missing information, etc; 2. OIS SNMP Monitoring will send out OIS SNMP Traps to itself, creating the infinite logical loop; 3. OIS Logs and Data Store grow quickly out of control with extraneous data.
Example Scenario 2:
Company B is using a 3rd Party Network Event Management System to Monitor not only Infrastructure SNMP Traps but other Network Events. Company B is also using OIS to Monitor the Events through that same 3rd Party System (to create Incidents, Change Requests, etc.). If OIS is set up with Self-Monitoring (SNMP Trap Destination = 3rd Party System), it sends out SNMP Traps when it encounters internal issues warranting an event. Any number of potential internal issues within OIS would result in an SNMP Trap being sent to the 3rd Party System. Since the 3rd Party System is also being monitored by OIS, OIS will pick up the 3rd Party Events it actually caused. If this is the case, three main problems occur: 1. Event Monitoring for all applications is in jeopardy, this means missing events, missing information, etc; 2. OIS will send out OIS SNMP Traps to the 3rd Party System, which in turn gets Monitored by OIS Event Monitoring (for the 3rd Party System), creating the infinite logical loop; 3. OIS and 3rd Party System Logs and Data Store grow quickly out of control with extraneous data.
Proposed Resolution:
Ensure there is a secondary or tertiary (non-OIS) SNMP Monitor that SNMP Traps originating from OIS can be captured/handled. If an outside SNMP Monitor system is used, that does not feed the primary system, the circular reference is broken and the risk of entering an infinite logical loop is removed.
REAL WORLD APPLICATION OF SELF-MONITORING
If the risk of an infinite logical loop can be avoided, the OIS Self-Monitoring functionality can be very useful in capturing and tracking event information throughout the life of a workflow (single or continued execution).
It is recommended that a 3rd Party Network Event Management System is used to capture the events in any environment, as this kind of software was specifically designed to perform this task. This way, the operators monitoring the 3rd Party System can take action based on all the combined information in the Infrastructure.
If a 3rd Party System is not available, and OIS is to be used as the Event Monitoring mechanism, there are some recommendations on how the information gets stored and forwarded to the appropriate recipients.
In the examples above, the information from the SNMP Traps is either stored in a Text Log file, or Database Table. These options are good for storage of the information coming from the OIS Self-Monitoring configuration, but there was no mention of what could be done with the information once it has been stored.
As mentioned above:
HINT: Regardless of which method is chosen to gather the information from the Self-Monitoring Event based SNMP Traps, once collected, the data is usually reported on and some kind of Notification is sent to an appropriate group. Reporting and Notification are not covered within this post, but OIS offers many methods for each. |
It is important to keep in mind that the information from these events is most time crucial to the sound running of the Infrastructure as a whole. While storing the information is good, getting the information to the appropriate people/groups is just as important.
OIS Policies can be built which execute at specified intervals, gather the latest information from the stored event data, and send email a consolidated report to specified people/groups. In turn, the people/groups receiving the notification and report can take the appropriate actions to get everything up and running smoothly again.
This kind of closed-loop remediation utilizing the OIS Self-Monitoring functionality is really useful in Real World Scenarios. It can offer relatively quick time-to-value and substantial ROI.
| SC ORCHESTRATOR 2012 NOTE: There is a System Center Operations Manager Management Pack on the System Center Orchestrator Roadmap which is due to release with the rest of System Center 2012 late in CY 2011. This MP replaces the need for much of this post’s content (if not all). |
Finally, as always, the content within this post falls into that “no warranty, no support, use at your own risk” category, and thus is for informational purposes only.
enJOY!