I will be back soon with some more content but things are fairly busy at the minute in our development cycle!  I wanted to share some news from our team (Windows File Server) about one of our technologies which is the Microsoft iSCSI Software Target.   This has been available for production use use as part of Windows Storage Server since early 2007. Storage Server is a version of Windows Server that is available only as an OEM branded appliance.  It has also been available for development and test use by MSDN and TechNet subscribers starting in May 2009. However, until now, there was no way to use the Microsoft iSCSI Software Target in production on a regular server running Windows Server 2008 R2. As of Monday this has been made publicly available to all users of Windows Server 2008 R2

Get all the details, including download instructions and FAQ at my one of my colleagues (Jose) blog:

https://blogs.technet.com/b/josebda/archive/2011/04/04/microsoft-iscsi-software-target-3-3-for-windows-server-2008-r2-available-for-public-download.aspx

This is very useful for the MP Author.  Why you ask?  Well using the iSCSI target you can easily set up a cluster in a single Hyper-V host with no external storage so you can test your MP behavior on a cluster.  This makes it much more accessible for folks.  I will follow up with a quick article on how to set this up.

Enjoy and kudos to our iSCSI Target team to get this delivered!

Let’s start with the basics of a failover cluster.  This post is not an introduction to clustering but I wanted to ensure you understand a little about network names and how these relate to computers in SCOM.  Testing against clusters can be a pain since clustering is not something you can readily set up in your environment.  However there are options here and I will follow up with a post on how to set up a cluster within Hyper-V if you have access to an iSCSI software target.

I am using a two node cluster.  To form this I took two domain joined Windows Servers with shared disk between them and followed the simple wizard to create the cluster.   When you create a cluster, the cluster name object (CNO) is created and you specify an IP address to use.  The screen shot below shows my basic setup:

So I have the following setup:

• stwilson7cl01, stwilson7cl02 are physical cluster nodes
• stwilson7cl is the cluster name object (this is a network name resource)
• Volume Q: is the shared disk I am using for the quorum witness disk

Now I am going to add a clustered File Server to the cluster using a second disk I have shared.  This is very easy to do using the Configure a Service or Application wizard from the failover cluster manager MMC.  This File Server adds the following resources to the cluster

• stwilson7clfs network name
• IP address
• Volume S:

This is shown below:

Now lets look at how this cluster appears in SCOM.  To monitor this cluster I did the following:

• Deployed agents to stwilson7cl01 and stwilson7cl02
• Set agent proxying to enabled on both nodes
• Waited for discovery to occur

If you go to the Windows Computer state view in the Operations Console you will see each network name resource in the cluster shown as a computer.   In my case I have two network name objects (one for the cluster name, one for the file server name).  Therefore I see the two cluster nodes (stwilson7cl01 and stwilson7cl02) and the two additional names (stwilsoncl and stwilsonclfs):

Notice that the virtual network names do not show an agent instance on them which is correct because they are actually monitored by the agents on the cluster nodes stwilson7cl01 and stwilson7cl02.

SCOM will manage with agent monitors the applications on each virtual name.  Basically the node that has the resource group active with the network name in it will do the monitoring.  SCOM handles the failover of this monitoring as the resource group moves between nodes.  The detail of this is not the purpose of this post however.

Cluster Ignorant Discovery

OK so why as an author do I care.  My application is not clustered so I don’t need to do anything – right?  Well actually there is a problem with this.  Lets look at the basic discovery for an application that is not cluster aware.  I am going to use a simple registry based discovery throughout these examples.

For the first example I am going to use a class called NonClusteredAppX in my MP (the MP is attached to this post).  I have targeted a discovery at the Windows Server class so this would run against all my Windows Servers which is a fairly normal practice.  Lets look at the discovery results using a simple state view I added to the MP:

Can you spot the problem?  Well I only have two nodes in the cluster but the application has been discovered on the virtual computer objects in the cluster since SCOM treats them as Windows Servers.  This is a problem which I will now show.  I have added a simple event monitor to my NonClusteredAppX class that looks for an event in the event log.  To see the full effect of the issue I made sure both resource groups (the cluster and the file server) are running on node one of the cluster.  Then I ran the following from the command line on node 1 to generate the event:

EventCreate –id 101 –t information –d “Test”

Now the state view shows this:

The problem is the monitor is being run by all instances of the application and you have one on each computer object.

If I had generated an alert from this monitor I would have 3 alerts for the same problem.   So as you can see this is something you might want to consider fixing.  If you application is indeed not cluster aware the fix is very simple and I cover that next.

If you are following along you can reset the state with this event:

EventCreate –id 102 –t information –d “Test”

Cluster Aware Discovery

There is a basic feature of the SCOM computer discovery we can take advantage of.  There is a boolean property defined on Windows Server called IsVirtualNode.  This will either be empty or true (never false).

For a non clustered application that can’t be clustered the basic fix to the discovery is to use this in the filter logic of the discovery.  In the MP I have added a new class called NonClusteredAppY.  For the discovery of this class I have added the logic below:

<Expression>
    <And>
        <Expression>
            <SimpleExpression>
                <ValueExpression>
                    <XPathQuery Type="Boolean">Values/ApplicationExists</XPathQuery>
                </ValueExpression>
                <Operator>Equal</Operator>
                <ValueExpression>
                    <Value Type="Boolean">true</Value>
                </ValueExpression>
            </SimpleExpression>
        </Expression>
        <Expression>
            <SimpleExpression>
                <ValueExpression>
                   <Value>$Target/Property[Type='Windows!Microsoft.Windows.Server.Computer']/IsVirtualNode$</Value>
                </ValueExpression>
                <Operator>NotEqual</Operator>
                <ValueExpression>
                    <Value Type="Boolean">true</Value>
                </ValueExpression>
            </SimpleExpression>
        </Expression>
    </And>
</Expression>

Now when this discovery runs, I only get instances created on the physical cluster nodes not the virtual computers.  Therefore instead of four instances I correctly get two:

So this may be all you have to do!  However, if you application is in fact designed to run on a cluster you need to swap this around a bit as the next sections show.

Cluster Only Application

If you want to only discover on a virtual computer in a cluster then you basically need to switch the logic around your expression as below:

<Expression>
    <SimpleExpression>
        <ValueExpression>
            <Value>$Target/Property[Type='Windows!Microsoft.Windows.Server.Computer']/IsVirtualNode$</Value>
        </ValueExpression>
        <Operator>Equal</Operator>
        <ValueExpression>
            <Value Type="Boolean">true</Value>
        </ValueExpression>
    </SimpleExpression>
</Expression>

My ClusteredAppA class in the attached MP shows this logic.  Now this will only be discovered on virtual nodes as shown below:

Note that it is likely you need to do more logic in your discovery since your application is probably not running in all resource groups.  I cover this at the end of this post.

Mapping IsVirtualNode

There is a slight extension to the cluster only application discovery above.  In my example MP, my classes are derived from Windows Computer Role.  This base class has a hosting relationship to Windows Computer not Windows Server.  This means that you can never get back to the fact that your instance is running on a cluster virtual computer object because if you tried to use a $Target/Host/Property expression it would resolve to the Windows Computer class that does not have this property.

Therefore if any subsequent discovery or monitoring that will be targeted at your class needs to have access to this property you should add a property to your application class and map this during discovery.  MY ClusteredAppB class shows this.  In the discovery I add a mapping:

<Setting>
    <Name>$MPElement[Name="AuthorMPs.Demo.ClusterDiscovery.ClusteredAppB"]/IsVirtualNode$</Name>
    <Value>true</Value>
</Setting>

Note that I se this to true since I know this is always the case since I have already checked in my expression for this.  If my application could run on clustered or non clustered I would do this instead:

<Setting>
<Name>$MPElement[Name="AuthorMPs.Demo.ClusterDiscovery.ClusteredAppB"]/IsVirtualNode$</Name>
<Value>$Target/Property[Type='Windows!Microsoft.Windows.Server.Computer']/IsVirtualNode$</Value>
</Setting>

Clustered and Non Clustered

The above showed some basics of how you can control cluster discovery and avoid the common pitfall.  What if you application can run on a cluster or non-clustered server.  In this case you are likely to need to do more than a simple registry discovery.  An example of this is SQL server.  During the discovery this checks if the current computer it is running against is clustered and also makes sure only instances that match this computer name are discovered on it.  The logic is fairly complex and is script based.  Most cluster aware application discoveries will be script based since each application has its own way of representing itself as clustered.

If you are using a script based discovery you should definitely consider the seed pattern I

AuthorMPs.Demo.ClusterDiscovery.xml

• By allowing monitoring to be changed at the source you potentially remove a trail of what went on.  If an application owners is screaming at Ops for missing an alert there is little trace of the fact that this was set by the same owner or the server administrator at the source
• Part of the benefit of SCOM in the enterprise is that it can centralize the administration of monitoring.  By following this you are potentially removing one of those benefits.
• Behavior in SCOM can be very different between monitored servers.  It can be hard to rationalize what monitoring is in effect as overrides affect some and not all monitored servers
• Using configuration at the source can lead to authors writing discoveries that are overly aggressive so that changes are picked up quickly.  This can lead to frequent instance space churn if properties are changing a lot.

There are times when this is what the customer wants so I wanted to illustrate one possible approach for this leveraging class properties and discovery from the registry.  There are some mitigations you can follow for the above as you go through this:

• Ensure only people authorized to make changes can actually make the changes.  Since you are using the registry in this example you need to be able to edit the registry which usually calls for administrative rights
• Consider a discovery mechanism such as an event to trigger a new discovery rather than using an aggressive polling frequency
• Limit the usage of this to key scenarios not all of your override cases

This is not something you can readily apply to sealed management packs.  It is really designed for home grown MPs for your custom monitoring needs as it involves defining new classes and discoveries.

Also note that this is designed for monitoring configuration overrides and not element level overrides such as enable / disable which wont work.  It is specifically tailored towards performance thresholds which is the example I use below.  However, any monitoring that takes an overrideable parameter could use this in theory.

So if you still want to do this, there are a few variations you can use and I will walk you through these and some of the pitfalls to watch for. 

The Basic MP

This is a very crude and simple example to show the concepts.  I am going to use a single class for each sample and use registry discovery to discover an instance when a registry key exists.  I am also going to populate a property on the instance with a value I get from the registry.  This is going to be used as a threshold in monitoring.  The actual monitoring is done using a simple two state performance threshold monitor.  For my example I am going to use the process count performance counter since this is very easy to change just by starting and stopping processes.

In the MP you will see that I use an abstract class as my base class and then I have defined 4 concrete classes from that.  This was just to save me time so I could just define the monitor once (targeted to the base class) and a single set of views.  Each approach uses a different class so you can see all the examples in the MP.  The classes used are:

1. Application X – used to show the non protected simplest approach
2. ApplicationX1 – a minor tweak to the first case to protect against a missing value
3. Application Y – uses two discoveries to provide a default value when the value is not set
4. Application Z – enhances the two discovery approach to provide an override to the default value

Disclaimer – as with all my sample MPs these are samples and should not be used in production.  Discovery intervals are aggressive and also the MP is not complete with lots of display strings missing etc.  Take this and incorporate into your MPs but use this in a test environment.

The Monitor

All the classes will have a monitor instantiated against them.  As I mentioned this is a simple two state monitor.  The configuration is shown below

<Configuration>
  <ComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</ComputerName>
  <CounterName>Processes</CounterName>
  <ObjectName>System</ObjectName>
  <InstanceName/>
  <AllInstances>false</AllInstances>
  <Frequency>60</Frequency>
  <Threshold>$Target/Property[Type="AuthorMPs.Demo.PropertyBasedMonitoring.ApplicationBase"]/ProcessThreshold$</Threshold>
</Configuration>

They key section is highlighted.  This tells the monitor to use the discovered ProcessThreshold value on the instance that it is instantiated for rather than a value defined in the monitor configuration.

Once you have one of the discoveries running below and an instance created you can test this out by ensuring your process count is higher than the threshold you set.  Once this happens the state will go red and you will get an alert.  Once it drops below the value the state returns to healthy and the alert is closed.

The rest of this post is about how we discovered this property.

Non Protected

This is the simplest case and is shown with the ApplicationX class and associated discovery.  I use a standard registry discovery that checks the existence of a key and also collects a value.  If the key exists an instance is created and the value is written as a property and you will see an instance similar to this:

You can easily try this out by creating the registry key:

HKLM\Software\AuthorMPs\ApplicationX

You should also create a DWORD value under that key called ProcessThreshold.  You should put an integer into this value that will be collected and stored on the discovered instance.  Once the instance is discovered you can go and test the monitor I describe above.

So that’s it right – we are done.  Well this might be OK but there is an important case to consider.  If the registry value was and optional setting that the administrator had to go and create it is very likely this is not set everywhere.  This causes a problem for our discovery and you can easily test this.  Go and delete the registry value ProcessThreshold but keep the key.  On the next discovery you will get a lovely event 10801 in the Operations Manager log indicating discovery data could not be inserted.  There is a lot of detail in this event but the main issue is called out as this:

Invalid monitoring class property value specified in the discovery data item.The value needs to adhere to the type.
MonitoringClassPropertyName: AuthorMPs.Demo.PropertyBasedMonitoring.ApplicationBase.ProcessThreshold
MonitoringClassPropertyValue:

You will also see the discovery data item trying to be inserted and notice that the property value is set to empty which is not a valid integer. 

Whether this is a problem depends on whether you can rely on the registry value always being there.  If you can’t you need to do a bit more work.  One option is to simply make the class property a string instead of an integer which will take an empty value.  However you are possibly pushing the problem downstream so if someone put a string in here your monitoring would fail.  Also your monitoring is going to fail on empty anyhow since the performance monitor will actually compare to a threshold of zero and always be in a bad state.  So quick fix is just avoiding the issue.

Whether you get an instance in your discovery will depend on the order of setting the registry value.  If the ProcessThreshold was never set, you will never get an instance since the discovery will fail.  If it was set and then removed later, the instance will remain but will never be undiscovered since all future discoveries are failing and will not generate a snapshot.  If you removed the registry key itself the instance would be removed.

Protected with Consequences

There is another fairly quick fix to the problem but it comes with consequences again which may or may not be an issue for you.

You can update the discovery to check that both the registry key and registry value exist.  You can do this easily with the following addition to the expression in the discovery:

<SimpleExpression>
  <ValueExpression>
    <XPathQuery Type="Integer">Values/ProcessThreshold</XPathQuery>
  </ValueExpression>
  <Operator>Greater</Operator>
  <ValueExpression>
    <Value Type="Integer">0</Value>
  </ValueExpression>
</SimpleExpression>

The ApplicationX1 class shows this in action.  It is exactly the same discovery as the ApplicationX class with this one change.  You will not get the discovery error above because we never try to write the value or instantiate the class unless the value is present and set to greater than 0.

The downside to this approach is that you must have this threshold set before you class is discovered.  If this value is being added just for a small subset of monitoring you want then you just messed up your other monitoring by not discovering unless the admin went and set this.  So think about this and see if that is an issue for your scenario.

So we have two solutions so far and neither one are comprehensive.  The next options look at setting a default value when there is nothing set in the registry.

Double Discovery

There is no easy way (at least that I came up with) to have a single discovery that has conditional logic where you could pull a value from the registry and if it was not there apply a default value instead.  You could easily do it with a script but I like to avoid scripts when possible.  Registry discoveries are lightweight and less error prone than scripts plus it is a challenge to sort this out.  A small additional OR gate module would easily allow this and I have asked for this to be added by the SCOM team before but with no success (it was there for a short period but then withdrawn before shipping!).

So instead we are going to use two discoveries.  These are registry discoveries and lightweight so adding one more is a minor overhead and these should not be running that frequency anyhow.  In the sample MP ApplicationY shows this double discovery method.

The first discovery is the same as ApplicationX1 above.  It will only discover if the key and value are present.  The second discovery (with the suffix DefaultValueNoOverride in the MP).  Is the interesting one.  This is very similar to the first discovery with some subtle changes.  First I collect the ProcessThreshold value as a boolean instead of an integer by changing the attribute type parameter:

<RegistryAttributeDefinition>
  <AttributeName>ProcessThreshold</AttributeName>
  <Path>SOFTWARE\AuthorMPs\ApplicationY\ProcessThreshold</Path>
  <PathType>1</PathType>
  <AttributeType>0</AttributeType>
</RegistryAttributeDefinition>

Next in the expression filter I match only when this is false i.e. the value does not exist.  Finally in the mapping configuration I set the ProcessThreshold to a value I specify in the MP e.g. 80 as shown below.

<Setting>
  <Name>$MPElement[Name="AuthorMPs.Demo.PropertyBasedMonitoring.ApplicationBase"]/ProcessThreshold$</Name>
  <Value>80</Value>
</Setting>

So the logic is:

• Registry key not present – both discoveries produces empty discovery data
• Registry key present but value not present – first discovery creates the instance, second discovery produces empty discovery data
• Registry key present and value present – first discovery produces empty discovery data, second discovery creates the instance

You can try these combinations out using the MP.  Note to discover Application Y use this key:

HKLM\Software\AuthorMPs\ApplicationY

So that must be it?  Well there is one problem with this solution.  I had to hardcode this default value in the MP buried in the configuration for the second discovery.  This makes it hard to change and it can’t be changed via an override.  Also if this MP was sealed you are stuck with this default.  So we can take this further into the night…

Double Discovery with Override

This is my last attempt I promise.  I want to allow the second discovery I used above to take an override that specifies the default value if the registry value is not found.  Unfortunately the only way to do this is to define my own composite module type to mimic the standard registry discovery module type that ships in the Windows library MP.

ApplicationZ in the MP shows this approach.  The first discovery is the same as the last example except I use the following key:

HKLM\Software\AuthorMPs\ApplicationZ

The second discovery uses a custom module type called AppZDiscoveryModule that takes two parameters – frequency and default value.

The AppZDiscoveryModule uses the standard registry discovery but critically makes both of its configuration parameters overrideable.  The main piece to note here is the mapping where I use the configuration parameter to set my default:

<Setting>
  <Name>$MPElement[Name="AuthorMPs.Demo.PropertyBasedMonitoring.ApplicationBase"]/ProcessThreshold$</Name>
  <Value>$Config/DefaultValue$</Value>
</Setting>

So I can now ship the MP with a default value but this can be overriden with a standard override.  You can find the override by locating the discovery in the Object Discovery view of the Authoring Space of the console.  When you override it you will see that DefaultValue is overrideable as well as frequency:

You can try this out by creating the ApplicationZ reg key above and not setting a ProcessThreshold value.  The instance should be discovered with 77 by default.  Now override the default to say 88 and wait for the configuration change to go down and the next discovery to run.  You should see 88 written.

To check that we still honor the real value when set, go and add the registry value and ensure that this is written in preference to the default value.

Note you could definitely write this composite module type better and reuse it between the two discoveries but this was just an illustration!

Conclusion

There is a lot of concepts here!  I just wanted to explore this solution a bit and show you that there are different levels of rigor you could approach this with.  You may be OK to stop at the first solution if you can guarantee the value is always there.  The choice is yours – have fun!

AuthorMPs.Demo.PropertyBasedMonitoring.xml

https://blogs.technet.com/b/operationsmgr/archive/2011/03/10/new-kb-installation-of-system-center-operations-manager-r2-authoring-resource-kit-fails-on-net-4-0-enabled-computers.aspx

First, make sure you have the MP Schema files.  You get these when you install the Authoring Console (latest version here).  The installation will create a MP Schema directory in your installation folder.  In this directory is the ManagementPacklSchema.xsd and other supporting files.  Note that if you copy these somewhere else you need all the files in the sub directories as well since the main schema file references this.

Next, ensure you have some version of Visual Studio 2010.  If you have an MSDN license you can get a full version from there or you company may be licensed for it.  The XML schema explorer ships with all versions except the Express versions. 

Once you have installed Visual Studio, run the program and open the ManagementPackSchema.xsd file.  You should get a view as as shown below by default.

If you do not see the XML Schema Explorer available on the right then click the link XML Schema Explorer to open it.   Locate the <>ManagementPack element in the list under the Empty Namespace, ManagementPackSchema.xsd in the tree view.

Drag this onto the design surface and you get an object that is fairly useless and does not look like you can do anything:

Now switch to the content model view which is the second tool bar icon from the left:

You should now see a workspace column on the left.  You need to drag the management pack element onto the surface if it is not already there:

Now you can expand the management pack element and drill into each area of the MP in a great visual way.  Below is an example of drilling into the ClassType element

If you want to jump to the XSD file to explore some more just right click on the element, attribute or type you want and select View Code and you will be taken in context to the correct place in the schema.

Update

Just adding to this post some information about getting the intellisense look up in the schema in Visual Studio when you are editing a Management Pack.  When I post this in the comments section the text is getting stripped out so posting it here.  Your MP declaration should be the following to point to the MP schema XSD.

<ManagementPack xsi:noNamespaceSchemaLocation="c:\MPSchema\ManagementPackSchema.xsd" xmlns:xsd="https://www.w3.org/2001/XMLSchema" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">

</ManagementPack>

Within SCOM you have two overarching choices for monitoring – state based and stateless.  Lets look at each and the benefits / drawbacks each one has and then I will get to the crux of things and why I think you should not try to force state based monitoring on something that you cannot really model the state.

Both of these approaches rely on some model of an application and discovery or that application and its components.  I am not going to focus on this end to end except where it can impact your choice of stateless or state based monitoring.  Also since notification can be used in both cases I will leave that out of the argument.

Stateless Monitoring

I will start with stateless.  In this model you are going to be using rules to generate alerts.  The model here is that a rule responds to an interesting event and raises an alert which to an operator.  The rule can optionally:

• Suppress or limit future alerts while the original alert is still open
• Run some action such as a script as well as generate the alert (note this cannot be added to a rule that was shipped in a sealed MP)

Rules are simple and have been with us a long time before any type of state based monitoring was provided (and I am sure will be with us a lot longer).  It is simple – I see something I care about I raise an alert.

As with all monitoring in SCOM, state based monitoring is targeted at a class.  You may or may not go to a granular level of monitoring with stateless monitoring and in many ways doing deep monitoring is less interesting since you are not going to be controlling state.  For example suppose I have a simple management pack and I model an application X with one or more components.  My instance space may look like this.

Now if I want to target some stateless monitoring for the App X class.  This is simple, I create some rules and target it at the App X class.  This will ensure that it only runs where I have discovered an instance of App X. 

For the App X component class I have a couple of choices with stateless monitoring.  I could either model things as above and target my rules at App X component.  I would then use some property of the App X component class as criteria for my monitoring so I can tell which instrumentation is for which instance e.g.

• Rule target: App X Component
• Rule criteria: Event 101 from event source Application X where Param1 is Component ID
• Suppression: Suppress on workflow

I will have at most one alert active for any instance of App X component although I could choose to have more by changing the suppression logic in my rule.

However since I am not monitoring the state of the Application X component you could ask what this deeper modeling is actually buying you other than more logic in your discovery.  You could equally simplify your model to this and do away with your component class:

Now you could create the same rule as follows:

• Rule target: App X
• Rule criteria: Event 101 from event source Application X where Param1 is Component ID
• Suppression: Suppress on Param1 (component ID)

As with the previous case I will still have at most one alert active for any instance of App X component at a time because I handled this in my suppression.  However, I have not modeled the App X component.

While the monitoring is effectively the same for this case there are other benefits to modeling to a deeper level.  Some of these are as follows:

• You can pivot to alerts / events / performance about a specific object in the console
• You can report on monitoring data at a more granular level
• You can create tasks against a component that use component properties to run

So you have the option of what you want to do and achieve.  The point about stateless monitoring is that you are not forced to monitor to a deep level.  Hold this thought as I will show you that with state based monitoring you may actually be forced to model deeper than you want.

With stateless monitoring, once an alert is generated, there is no understanding of when the problem is resolved.   A user (or automated system like a connector) must close the alert.  Depending on suppression settings of individual rules, new alerts may or may not get generated until this is done.  You are forcing the diagnosis and resolution state of problems to the operator of the system which may indeed be valid if you application has no understanding of it’s own state.

State Based Monitoring

With state based monitoring you are using monitors as your primary method.  As you likely know monitors are very different to rules.  Some basic characteristics of normal monitors:

• Monitors have two or three states
• States can be automatically determined (instrumentation / timer) or manually set
• Monitors can optionally alert based on state
• Monitors can optionally close alerts based on state
• A maximum of one alert from a single monitor can be active
• Monitors can roll up to other monitors

When thinking about state based and stateless monitoring choices there are a few areas I want to focus on.

The idea of a monitor is that at any point of time it should know the state of a part of your model.  There should be no doubt.  When a monitor shows red in the console it should mean there is a problem right now.  This is very different to an alert from a rule that shows at some point there was a problem.  While this idea sounds great, in practice there are a number of monitor types that break this concept:

• Timer based reset monitors – the unhealthy state of a monitor can be detected but there is no way to detect a good state.  Instead a timer is set so that the monitor will reset after a specified period of time.
• Manual reset monitors – the unhealthy state of a monitor can be detected but there is no way to detect a good state.  The user must manually reset the monitor when this is known to be true.

To me this breaks the whole concept of state based monitoring and using these types of monitors should have a very good reason (there are a few).  You don’t understand the state of the system at any given time and instead you are making a prediction that the problem may have gone away in one case and in the other you are forcing a user to manually intervene.  Logically both of these should probably have been done by a stateless rule.  Monitors do have some benefits over rules though:

• You can report on availability over time
• You can populate state views in the console
• You can run diagnostics and recovery workflows based on a state change
• Diagnostics and recoveries are extensible by customers even against sealed MPs
• You can roll up health through a health model on the same object or across discovered relationships

So when you think about monitors think about whether the benefits of the above outweigh the issue you are not accurately representing the state and you should ensure customer expectations are set on the behavior of these monitoring.

To be honest I have seen lots of examples of state based monitors that use manual / timer reset monitors that could actually be rewritten to properly determine the good state.  This may involve some sort of polling of system state for the good state.  While this may require the definition of a new monitor type and a bit more development time, the benefits to the customer definitely outweigh the effort it will take you to do this.

In terms of modeling, using monitors may force you to model classes deeper than you want.  This may not be a bad thing but at some point you will stop.  For example when monitoring a database server you may want to model down to the database level but modeling to the table or even row level with classes is not where you want to go.  Using the example above, let’s consider the simple model:

Now I want to alert before on the App X component but assume I do not want to model this level in my application.  If I know 101 is the good event and 102 is the bad event you might assume you can do the following and create a monitor:

• Monitor type: 2 state event based
• Monitor target: App X
• State 1 (unhealthy): Event 101 from event source Application X
• State 2 (healthy): Event 102 from event source Application X

Monitors cannot define alert suppression since you get a maximum of one alert form a given monitor.

This monitor will have a major problem.  If you have two components (Comp1 and Comp2) consider this:

• Comp1 throws event 101 – monitor goes unhealthy
• Comp2 throws event 101 – no state change since monitor is already unhealthy
• Comp2 throws event 102 – monitor goes healthy

In this flow we are broken.  Comp1 is still in a bad state and we are showing healthy for Application X.  You may think this is obvious but I have seen this many times in MPs and this is what drove me to write this post.

If you are using a monitor you have to be able to aggregate what you are doing to the class the monitor targets.  In the above example you would want to consider how you would know across all components whether App X is health or not healthy.  You may not be able to do this using events.  The other option is of course to model deeper:

Now the monitor would be defined as follows:

• Monitor type: 2 state event based
• Monitor target: App X Component
• State 1 (unhealthy): Event 101 from event source Application X where parameter 1 matches Component ID
• State 2 (healthy): Event 102 from event source Application X where parameter 1 matches component ID

Optionally you might define a dependency monitor on the App X class to roll up health from the components.  Now the monitor targeted at the component class above is just responsible for a single component and uses event parameters to filter out events about other components.

Note that monitors have another characteristic to be very aware of when you are choosing between a monitor and a rule.  The alert generated from a monitor is effectively separated from the monitor state.  Subsequent changes to the alert will not affect the monitor state.  The major concern here is when the alert is resolved in the console.  This act will not currently reset the monitor at all.  If the monitor was unhealthy and the alert is resolved, the monitor will still be unhealthy until the healthy event is seen.  Critically this means no future alerts will be generated from this monitor instance until the thing it is monitor is either manually reset using health explorer or the object returns to a health state itself and then goes back to a bad state.  Using manual reset monitors make this worse – once a user resolves the alert it is never coming back till someone goes and resets the monitor which is a very bad experience.

Conclusion

So should you use state based or stateless monitoring?  The answer is both.  There are reasons to use both of them and the day we can get away from stateless monitoring are a long way away.  As a result you are very likely to be using both in your management packs.  My advice would be:

• Make sure you understand the capabilities and differences of monitors / rules
• Model to an appropriate level in your management pack – don’t ship hundreds of classes that are not of interest to customers just so you can use monitors as you want to
• Use monitors where you can accurately determine all states via instrumentation
• Use rules when you cannot determine state but there are important alerts to raise
• Use rules or aggregated monitors to monitor levels of an application you do not want to model down to
• Limit your use of manual reset / timer based reset monitors.  When you use them be sure you understand the impact to customers

You should definitely strive for state based monitoring where possible but my key message out of all of this is not to be afraid of using rules still.  Do not force state on something that is inherently stateless. 

Finally, if you own the application you are building a management pack for then think about how you can improve your instrumentation to move towards being able to use monitors more by understanding and exposing application state better.

Let’s start by looking at a standard discovery with no seed concept. Let’s say you have an application running on Windows and you want to define a class for the application and discover it. Let’s assume that this runs on any version of Windows Server. You would define your class something like the following:

<ClassType ID="AuthorMPs.Demo.ApplicationX" Abstract="false" Accessibility="Internal" Hosted="true" Base="Windows!Microsoft.Windows.LocalApplication"/>

Now if you want to discover this using a simple registry based discovery you might create a discovery targeted at Windows Server

<Discovery ID="AuthorMPs.Demo.ApplicationX.Discovery" Target="Windows!Microsoft.Windows.Server.Computer" Remotable="false" Enabled="true">
  <!--Registry based discovery Logic in here-->
</Discovery>

This all seems reasonable so far. What will happen in this case when this MP is imported into your management group is that the MP will be sent to every computer that has an instance of the Windows Server class discovered i.e. it will go to every Windows Server you have. Then the discovery workflow will be loaded and run on each server.  This discovery will stay loaded and keep running on whatever interval you set in your registry discovery even if the application is not found.  This is done because the application could be installed at any point and then you want SCOM to pick it up.  In this example you would end up with a single instance of the class hosted on servers that had the application installed as below:

For a registry based discovery this is not a major concern.  SCOM is very efficient at running these discoveries and they are usually not schedule to run more than every few hours so it is very little overhead to the server it runs on.

However, the problem comes when you need more than a registry based discovery to discover the application fully.  It may be that you need a complex script to discover the application including the properties you need and any components.  In this situation you need to be careful especially if you are building MPs to ship to customers.  You want to minimize the execution of scripts on servers where possible and running a full blow discovery script on a scheduled interval is likely overhead you can avoid.  This is where the seed concept comes in.

The seed concept involves introducing a second class that acts as the primary discovery.  This will not result in the full discovery of the application and is only there to trigger a second stage discovery that will do the real discovery.  For example we may introduce a seed class as below:

<ClassType ID="AuthorMPs.Demo.ApplicationXSeed" Abstract="false" Accessibility="Internal" Hosted="true" Base="Windows!Microsoft.Windows.LocalApplication"/>
<ClassType ID="AuthorMPs.Demo.ApplicationX" Abstract="false" Accessibility="Internal" Hosted="true" Base="Windows!Microsoft.Windows.LocalApplication">
  <Property ID="Version" Type="string"/>
  <Property ID="SomeProperty1" Type="string"/>
  <Property ID="SomeProperty2" Type="string"/>
</ClassType>

Now I build my first discovery to identify that the application is present on the system with a simple registry based discovery.  This is my seed discovery and is targeted at Windows Server in this example:

<Discovery ID="AuthorMPs.Demo.ApplicationX.SeedDiscovery" Target="Windows!Microsoft.Windows.Server.Computer" Remotable="false" Enabled="true">
  <!--Registry based discovery Logic in here-->
</Discovery>

If this seed was discovered it indicates my application is there.  However I am not ready to start monitoring yet.  I need to discover the application fully and I need a complex script to do this.  So I now add a second discovery that is targeted at the seed class.  This will only run once I have validated the application is present.

<Discovery ID="AuthorMPs.Demo.ApplicationX.Discovery" Target="AuthorMPs.Demo.ApplicationXSeed" Remotable="false" Enabled="true">
  <!--Script based discovery Logic in here-->
</Discovery>

This discovery will do whatever complex scripting it needs to do to discover one or more instances for the application.  From here on things are normal.  So the flow is:

All your actual monitoring should be targeted at the real class not the seed class.  This monitoring will only start working after the second discovery has run and discovered the instance with any properties that are required.

The artifact of this process is an additional class and an instance of that class on every server the application is discovered on.  This is not a major problem but there are a few things you can do to ensure customers are not confused by this:

• Ensure the word seed is in the display name for this class
• In the class description explain what it is used for and indicate no monitoring should be targeted at the class
• Do not ship a view that shows state, alert etc. about this class
• In your MP guide if you have one make sure you detail this class and how it is used

If the application is uninstalled, the behavior of removing instances depends on which discovery runs first after uninstall:

• If the seed runs first, it will submit an empty snapshot for the seed class and this instance is deleted. However because the real instance has this seed instance as a source, SCOM will also remove it at the same time since its discovery source no longer exists. Both instances in the example above are removed effectively at the same time
•  If the second discovery runs first then this will submit an empty discovery for the real class and this instance will be removed. The seed instance still exists and this will only be removed on the next schedule of the seed discovery so it is a two stage process

You could definitely consider the seed based discovery for scripts but you might want to consider for WMI based discoveries as well if the WMI queries you are using are complex and may return many objects.  If you absolutely cannot avoid using a script as there is no reliable way of detecting the application through the registry you should write your script that it exits at the earliest opportunity once it determines the application is not installed.

There is one alternative approach that sometimes people consider.  You could have a single class that is partially discovered by multiple discoveries:

• Discovery 1 is really the seed discovery and discovers existence of the application but does not populate all properties
• Discovery 2 runs a script and discovers all the missing properties and any other classes it needs to for the application

This is illustrated below.  In this example Application X has 4 properties.  Two are discovered by the base discovery which is a registry discovery.  Two are discovered by the full discovery which runs after the instance is created.

This would be built with a single class:

<ClassType ID="AuthorMPs.Demo.ApplicationX" Abstract="false" Accessibility="Internal" Hosted="true" Base="Windows!Microsoft.Windows.LocalApplication"/>

The first discovery targets Windows Server:

<Discovery ID="AuthorMPs.Demo.ApplicationX.BaseDiscovery" Target="Windows!Microsoft.Windows.Server.Computer" Remotable="false" Enabled="true">
  <!--Registry based discovery Logic in here-->
</Discovery>

The second discovery targets the Application X class and discovers the additional properties:

<Discovery ID="AuthorMPs.Demo.ApplicationX.FullDiscovery" Target="AuthorMPs.Demo.ApplicationX" Remotable="false" Enabled="true">
  <!--Script based discovery Logic in here-->
</Discovery>

In theory this works OK but there are some problems:

• Monitoring may start to run before the second discovery completes. If this monitoring uses properties P3 and P4 in this example it will fail since these are not discovered yet.
• If the second discovery fails for some reason which is more likely in a script that a simple registry discovery, you are in a half discovered state and again monitoring may fail as above
• There is some caching behavior that can cause a transient issue:
  • Application Y is discovered by the seed and P1, P2 are populated
  • Second discovery runs and P3, P4 are discovered
  • Now Application Y is uninstalled and the object is removed as expected
  • Application Y is reinstalled
  • Application Y is discovered by the seed and P1, P2 are populated again
  • P3 and P4 are set to the previous value discovered in the last run. These will be correctly discovered in the next run of the full discovery but you can end up with a transient issue here

The caching problem above is due to the fact there was valid data still available for the second discovery.  As soon as the instance came back this data was still there and will remain till a new discovery runs.  If you want to see this happening for yourself in the sample MP follow these steps: 

• The Application Y full discovery adds a property to the instance of the number of sub folders on the C:\ folder.
• Get it discovered by adding the appropriate registry key – check the count gets added
• Now rename the registry key so App Y gets undiscovered.
• Add a folder to C:\
• Disable the full discovery to make sure it does not run again (leave the seed enable)
• Rename the registry key back to App Y gets discovered
• Observe that the old value initially for the directory count appears even though the discovery was disabled
• Enable the discovery again
• You should see the new value added
• If you can’t see this try disabling the second discovery before you rename the registry key back so it never runs again and you will see we have kept the old data of the property values.

I have attached an MP showing both concepts.  Application X shows the seed concept and Application Y shows the multiple discovery concept not using a seed:

• For Application X discovery, add a registry key HKLM\Software\AuthorMPs\ApplicationX and a string value HKLM\Software\AuthorMps\ApplicationX\Version and set it to something.
• For Application Y discovery, add a registry key HKLM\Software\AuthorMPs\ApplicationY and a string value HKLM\Software\AuthorMps\ApplicationY\Version and set it to something.

So bottom line, while the second method reduces the number of classes and means there are no seed instances, in practice it is much preferable to use the first method for your discovery when you are using scripts to avoid transient issues.

AuthorMPs.Demo.SeedDiscovery.xml

As I was moving content over from the old authormps, I was reviewing all the old schema documentation I had up there. This is really duplicated by the MSDN documentation which came after I started the site. As a result rather than porting this over I am just going to point to MSDN instead. If there is anything not here then use the feedback comments on the MSDN site and it will get routed to the writers or post a comment to the news groups.

You can find the schema documented here:

https://msdn.microsoft.com/en-us/library/ee533489.aspx

If you actually want the schema files to author against, you need to download the R2 authoring resource kit.  After you install this you will find the files in the “MP Schema” directory of your install.  You can get the resource kit here:

https://www.microsoft.com/downloads/en/details.aspx?FamilyID=9104af8b-ff87-45a1-81cd-b73e6f6b51f0&displaylang=en

Every workflow in Operations Manager is targeted at a type of object or a class. The workflow will only be run where an object of this type has been discovered and is being managed by Operations Manager. A separate workflow will execute for every object that is present of this type.

Rule

A rule uses data source, condition detection and write action module types in its definition. A rule will be loaded into memory as soon as an object of the type that the rule is targeting is managed by the healthservice. A rule continues to be held in memory waiting for data items to be generated from the specified data source(s). A rule must be formed according to the following flow chart:

A rule consists of one-to-many data source modules, zero or one condition detection module and one to many write action modules. If more than one data source module is used, a condition detection module must be used and the module type of the condition detection module must accept the same number of inputs as the number of data sources present.

Each module in the workflow is defined to be of a particular module type. The module type defines the behavior of the module and may be a code based implementation or a composite of other module types.

Every rule must end in at least one write action that changes the monitored system state or Operations Manager state e.g. raises an alert, stores data in the database.

A rule is typically used for one of three purposes:

• Collect data and store it in the Operations database and/or the data warehouse

• Generate an alert based on instrumentation observed

• Run a timed action on a continuous basis e.g. run a script every five minutes

Discovery

A discovery is a specialized type of rule. A discovery is used solely for the purpose of inserting discovery data into the Operations Manager database. This discovery data may be object instances and/or relationship instances. The workflow for a discovery consists of a single data source:

No condition detection modules or write action module types are allowed. There is an implicit write action for all discoveries that is handled by Operations Manager internally. This write action inserts the discovery data into the database. Operations Manager also handles discovery data caching internally so data is only sent to the management server and database if it has changed from the last execution of the discovery.

The module type that is used for the data source must output System.DiscoveryData in order to be used in a discovery workflow. If you try to use a module type that does not output this type of data, MP Verify will not pass for you management pack.

A discovery for each instance of the object type that the discovery is targeted at is loaded and held in memory waiting for data items to be generated from the specified data source.

Although a discovery can only contain a single data source module, a data source module type can be composed of other modules. For example a scheduler can be composed with a script probe action into a new data source module type. Many of the out-of-the-box data sources that are defined for discovery purposes are composite module type definitions.

Task

A task is a workflow that is not loaded by the health service until it is requested by the user through the Operations Console or some other SDK call. The user executes the task for one or more objects, the agent receives notification from the server to execute the workflow and then output returned to the database. Once the workflow has executed, it is unloaded from memory until it is called again. A task does not have a data source or a condition detection module. It must consist of either a single probe action or a single write action. You should use the correct type of module depending on what your task will do. If it is only reading system state, use a probe action. If it is changing system state in some way, use a write action.

You will recall that probe actions and write actions expect some input data. This is provided for a task in the form of a trigger signal by the health service. As a result you should only use a probe action or write action module type in a task when the type expects System.BaseData or trigger only data. For example if your module type is expecting performance data or event data, it is not going to get this data type when it is launched as a task.

Monitor

A monitor is used in Operations Manager to determine state of a monitored object. Each monitor is a distinct state machine that has a set number of states. A monitor is in a single state at any one time. Aggregate and dependency monitors are specific types of monitors that derive their state from other monitors only. Although these use workflows internally it is not necessary to understand the internals of these types for this discussion. This section focuses on unit monitors only.

A unit monitor is defined to be or a particular monitor type. The monitor type defines the workflows that the monitor will use. Typically a monitor type has a separate workflow for each state that the monitor type declares. In addition on-demand workflows can be defined for each state as well. Therefore when a monitor is created there can be many monitors running for each instance of the monitor.

For regular detection workflows (normally ongoing operation of a monitor) any number of modules can be used. The rules for the workflow are:

• The workflow must begin with one or more data source modules

• The workflow must not use write action modules

• The workflow can use condition detection and probe actions modules in any order

• Only one data stream should be outputted i.e. if you are using two data source modules you must pass the output to a condition detection module

For each health state of the monitor, the workflows should be mutually exclusive i.e. data should only be output for one workflow for a given set of instrumentation. Since a monitor is a state machine it is invalid to output both states at a given time. Therefore when you are using an expression filter in your workflows you should make sure both expression cannot evaluate to true at once.

As an example a two state monitor may define the following workflows for regular detection:

State 1:

State 2:

It is possible that one monitor state can declare no workflow for regular detection. This means that this state can only be entered when the user resets the monitor or the monitor is reset by the health service (comes out of maintenance mode or initiates for the first time)

The on-demand workflows for a monitor type define what state will be set when the monitor is reset (by the user, by exiting maintenance mode or on first initialization). These workflows must follow these rules:

• The workflow must begin with a probe action

• The initial probe action must be trigger only i.e. it takes no input data

• The workflow can use condition detection and probe actions modules in any order

• No data source or write actions can be used in the workflow

A two state monitor may define the following workflows for on-demand detection:

State 1:

State 2:

As with regular detection you should ensure only one state evaluates at a given time to avoid non deterministic behavior.

Diagnostic

A diagnostic is a specialized type of task. A diagnostic uses a monitor state change to trigger execution. It is either executed on demand by the user (or another SDK call) when a monitor is in a given state or automatically by the health service when the state is entered. Diagnostic workflows are targeted to a type of object, a monitor and a specific health state of the monitor. A diagnostic is not loaded by the health service until it is required.

The diagnostic consists of zero or one condition detection modules and one probe action module. A diagnostic should not change system state in any way.

The condition detection will often be a filter that looks at the input data item (the state change with the original data item that caused the state change as context) and evaluates it against a filter.

Multiple diagnostics can be created for the same monitor state if required.

Recovery

A recovery is another specialized type of task. A recovery uses a monitor state change or diagnostic output to trigger execution. It is either executed on demand by the user (or another SDK call) when a monitor is in a given state or automatically by the health service when the state is entered. Alternatively it may execute only after a specific diagnostic has provided output data. Recovery workflows are either targeted to a type of object, a monitor and a specific health state or they are targeted to a type of object and a specific diagnostic that is targeted to the same type.

The recovery consists of zero or one condition detection modules and one write action module. A recovery changes system state is some way.

The workflow for a recovery triggered by a state change is:

The workflow for a recovery triggered by a diagnostic is:

Operations Manager uses modules to define workflows. Workflows are defined in management packs and distributed to agents and servers via the Operations Manager channel. Workflows can do many things such as collect information and store in the operations database or data warehouse, run timed scripts, create alerts, run on-demand tasks etc. Workflows are defined using modules and modules are defined to be of a particular type known as a module type. A module type can either be a code implementation (native or managed) or a composite of other module types. There are four different module types that can be defined. Each of these has different semantics.

Data Source

A data source module type generates data using some form of instrumentation of some timed trigger action. For example a data source may provide events from a specific Windows event log or it could poll Windows performance counters every 10 minutes for a specific performance counter. There are many data sources available in Operations Manager.

A data source takes no input data and provides one output data stream. The module type will normally require some configuration to specify the behavior e.g. computer name and log name in the case of the Windows event log data source. The module can be represented as follows:

Probe Action

A probe action module type uses some input data to provide some output data and will usually require some configuration to specify the behavior. A probe action will interrogate a monitored entity in some way but it should not affect system state in any way. An example would be running a script that used an application API to generate some data or querying WMI to get some data. Often a probe action will be used in conjunction with a scheduler data source to run some action on a timed basis. The probe action module type may or may not use the input data item to affect the behavior. If the data item is not used the input data should be defined as trigger only. An incoming data item will trigger the probe to execute but the data is not used. Sometimes the data may be used for example if the probe is triggered by a Windows event, it may be required to pass the event ID to the probe action as configuration. A probe action can be represented as:

A probe action provides a single output data stream.

You should note that Operations Manager cannot determine if the probe action is being used to change system state in some way e.g. if you run a script defined using a probe action module type, you could be changing state in some way in your script. It is up to the MP author to adhere to the module type definition guidance. If you are changing system state you should use a write action module type instead.

Condition Detection

A condition detection module type is used to filter the incoming data in some way. There are many types of filter possible e.g. a simple filter on the input data, consolidation of like data items, correlation between multiple inputs, averaging and optimization of performance data etc. A condition detection module type can take one or more input data streams and provides a single output data steam. It can be represented as follows:

Many of the data sources that ship out of the box will already contain an expression filter in their composite definition so it may not be necessary to use a condition detection module in your workflow definitions.

Write Action

A write action module type takes a single input data stream and uses this in conjunction with some configuration to affected system state in some way. This change could be in the monitored system or in Operations Manager itself. For example this may be to run a script that changes something, write data into the Operations manager database or generate an alert. A write action can be represented as:

A write action may or may not output data. This data cannot be passed to any other module because the write action is the last module in a workflow. However, the data may be sent to the Operations database. An example is running a command that outputs data to standard out. This data may be useful to the operator that executed the command and is returned to them and stored as task output.