The System Center Operations Manager Support Team Blog

The System Center Operations Manager blog has moved!

J.C. Hornbeck — Tue, 14 Feb 2012 09:41:00 GMT

With the impending release of System Center 2012 Operations Manager, we thought now would be the perfect time to consolidate our blog properties and give you a single, concise gateway into the world of all that is Operations Manager. No longer will we have one blog by the product team and another by product support, but now we’ll have a single blog where everyone involved with developing, deploying, using and supporting Operations Manager can come together and contribute. We’re hoping that this will make it easier for you to find all the information you need, all from a single source.

So what does this mean to you? Starting today we will no long be publishing new articles to https://blogs.technet.com/b/operationsmgr. All of the existing articles will remain right where they’ve always been, but new content will instead be posted to our Operations Manager team blog at https://blogs.technet.com/b/momteam. This means that if you have a current RSS subscription to to https://blogs.technet.com/b/operationsmgr you’ll need to update it to point to https://blogs.technet.com/b/momteam/ instead. If you follow us on Facebook or Twitter nothing will really change so the transition will appear pretty seamless.

We have a lot of exciting things planned so be sure and stay tuned!

J.C. Hornbeck | System Center & Security Knowledge Engineer

Get the latest System Center news on Facebook and Twitter:

The Forefront Server Protection blog: https://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog : https://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : https://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: https://blogs.technet.com/b/isablog/
The Forefront UAG blog: https://blogs.technet.com/b/edgeaccessblog/

New Application Performance Monitoring (APM) Forum

J.C. Hornbeck — Thu, 09 Feb 2012 12:30:14 GMT

We have created a new forum under the Operations Manager forum category on TechNet for Application Performance Monitoring (APM). Please feel free to fire away questions to Microsoft, the MVPs, and the broader community, answer questions, and have some good discussions.

Here’s the link:

https://social.technet.microsoft.com/Forums/en-US/scomapm/threads

Release Candidate of System Center 2012 Integration Packs for Orchestrator Released!

J.C. Hornbeck — Wed, 08 Feb 2012 12:23:00 GMT

Following the release of System Center 2012 Release Candidate, Microsoft is pleased to announce the availability of the Release Candidate of the Orchestrator Integration Packs for the following System Center 2012 components:

System Center 2012 Virtual Machine Manager
System Center 2012 Operations Manager
System Center 2012 Data Protection Manager
System Center 2012 Service Manager

In addition, Release Candidate Orchestrator Integration Packs are also available for the following legacy System Center products:

System Center Operations Manager 2007 R2
System Center Service Manager 2010
System Center Virtual Machine Manager 2008 R2
System Center Data Protection Manager 2010
System Center Configuration Manager 2007

All of the above listed Integration Packs allow customers and partners to gain better automation and integration across their datacenter and private cloud environments by standardizing IT process and systems activities to drive greater consistency and improve operational ROI.

To obtain the Release Candidate Integration Pack download, visit https://www.microsoft.com/download/en/details.aspx?displaylang=en&id=28725

System Requirements:

System Center 2012 Component Support: System Center 2012 RC Orchestrator must be installed prior to installing the System Center 2012 RC Integration Packs.

Installation Instructions:

System Center 2012 RC Orchestrator Integration Packs

Download the System_Center_2012_Orchestrator_Integration_Packs.EXE file from the link above
Double click on System_Center_2012_Orchestrator_Integration_Packs.EXE
Follow the on screen prompts to extract the Integration Packs
After you extract the Integration Pack files, you must register them with the System Center 2012 RC Orchestrator management server and then deploy it to Runbook Servers and Runbook Designers. For instructions on how to install Integration Packs, see https://go.microsoft.com/fwlink/?LinkID=239936.

Related Resources:

Integration Pack Installation Instructions https://go.microsoft.com/fwlink/?LinkID=239936

System Center 2012 https://go.microsoft.com/fwlink/?LinkId=216991

Integration Pack Guide https://go.microsoft.com/fwlink/?LinkId=220929

This Release Candidate contains the following:

Orchestrator Integration Pack for System Center 2012 Virtual Machine Manager
- 30 Runbook Activities including: Create Checkpoint, Create New Disk, Create New
  Disk from VHD, Create Network Adapter, Create User Role, Create VM From
  Template, Create VM from VHD, Create VM from VM, Get Checkpoint, Get Disk, Get
  Network Adapter, Get User Role, Get User Role Quota, Get VM, Manage Checkpoint,
  Move VM, Remove User Role, Remove VM, Repair VM, Resume VM, Run VMM PowerShell
  Script, Shut Down VM, Start VM, Stop VM, Suspend VM, Update Disk, Update
  Network Adapter, Update User Role Property, Update User Role Quota and Update
  VM.
- RunVMM PowerShell Script allows runbooks to easily interact with the SC2012
  VMM PowerShell surface.

Orchestrator Integration Pack for System Center 2012 Service Manager
- 13 Runbook Activities including: Create Change with Template, Create
  Object, Create Incident with Template, Create Related Object, Create
  Relationship, Delete Relationship, Get Activity, Get Object, Get Relationship,
  Monitor Object, Update Activity, Upload Attachment and Update Object.

Orchestrator Integration Pack for System Center 2012 Operations Manager
- 8 Runbook Activities including: Create Alert, Get Alert, Get Monitor,
  Monitor Alert, Monitor State, Start Maintenance Mode, Stop Maintenance Mode and
  Update Alert.
Orchestrator Integration Pack for System Center 2012 Data Protection Manager
- 9 Runbook Activities including: Create Recovery Point, Get Data Source,
  Get Recovery Point, Get DPM Server Capacity, Protect Data Source, Recover
  SharePoint, Recover SQL, Recover VM and Run DPM PowerShell Script
- Run DPM PowerShell Script allows runbooks to easily interact with the SC2012
  DPM PowerShell surface.
Integration Packs for System Center Virtual Machine Manager 2008 R2, Operations Manager 2007 R2, Configuration Manager
2007, Service Manager 2010, Data Protection Manager 2010 (unchanged from previous RC).

(YES!) Start/Stop Maintenance Mode for a Windows or Unix computer now recurses
through all child monitors on that computer!
New published data from Get Alert and Monitor Alert:CompanyKnowledge, ManagementPackMonitorName,
       MonitoringRuleName, ManagementPackName, ManagementPackFriendlyName,
       ManagementPackDisplayName, ManagementPackId, ManagementPackVersion,
       ManagementPackIsSealed

Post RC:

Integration Pack for System Center 2012 Configuration Manager

Operations Manager Dynamic Group Examples

J.C. Hornbeck — Thu, 02 Feb 2012 10:02:46 GMT

Here’s another cool article by Microsoft’s own Jeanie Decker that I found over in our TechNet Wiki. This one goes over some examples of dynamic groups in System Center Operations Manager.

Do you have examples of dynamic group formulas that you want to share? Consider adding them to this article, or just link to examples you've already published.

In Operations Manager, groups are logical collections of objects, such as Windows-based computers, hard disks, or instances of Microsoft SQL Server. Groups are populated by explicitly adding objects to the group or dynamically according to criteria you set. For more information on the use of groups, see Creating and Managing Groups in the Operations Guide.

This article provides example of group definitions. The examples describe the items to select in the Query Builder and the resulting formula…

You can read Jeanie’s entire article here.

J.C. Hornbeck | System Center & Security Knowledge Engineer

Get the latest System Center news on Facebook and Twitter:

The Forefront Server Protection blog: https://blogs.technet.com/b/fss/
The Forefront Identity Manager blog : https://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: https://blogs.technet.com/b/isablog/
The Forefront UAG blog: https://blogs.technet.com/b/edgeaccessblog/

System Center 2012 Operations Manager incorrectly reports that a restricted System Center 2012 Data Protection Manager action completed successfully

J.C. Hornbeck — Tue, 31 Jan 2012 10:12:26 GMT

Here’s a new Knowledge Base article we published today. This one talks about an issue where a restricted user can attempt certain actions that they don't have permission to yet get a (false) message indicating it was successful.

=====

Symptoms

When using System Center 2012 Operations Manager (OpsMgr) to manage System Center 2012 Data Protection Manager (DPM) servers and leveraging the role-based access control, it is possible that a restricted user can attempt certain actions that they don't have permission to and get a message indicating it was successful.

Checking the jobs view in DPM for the given protection group will show that the job is not actually initiated in DPM. Restrictions for the user will be followed.

This is most likely to happen only when dealing with a protection group. When the same action is attempted at the data source level, users will be presented with the expected message indicating the action is not allowed.

Cause

This is caused by unexpected return values from the agent task in OpsMgr.

Resolution

The DPM team is aware of the condition and is working on a resolution that will be targeted for the next DPM QFE.

It is important to note that the action is not actually being performed. There is no unexpected elevation of privileges. The roles as defined in OpsMgr are being honored at all times.

More Information

Following are examples of the expected behavior and the erroneous message.

If a restricted user attempts to launch a consistency check for a data source from the OpsMgr console, initially a message will be displayed indicating the job is being sent.

Run Consistency Check on data source
Status: Run Consistency check on data source

This will then be followed by an error indicating the user does not have permission.

Failed to start on data source
You do not have permissions to perform this action. Your DPM administrator must give you permissions to any one of the following tasks - Run (ID: 33238)

If the same user attempts to initiate the consistency check, but this time at the protection group level, the experience changes and an erroneous message is displayed. First, an indication that job is being sent will be displayed.

Run Consistency Check on Protection Group
Status: Run Consistency Check on Protection Group

At this point, OpsMgr reports that the job was successfully started.

Run Consistency Check on Protection Group
Status: Successfully started consistency check on Protection Group

The above message is the erroneous message as the job will not be started on the DPM server.

=====

For the most current version of this article please see the following:

2665983 : System Center 2012 Operations Manager incorrectly reports that a restricted System Center 2012 Data Protection Manager action completed successfully

J.C. Hornbeck | System Center & Security Knowledge Engineer

Get the latest System Center news on Facebook and Twitter:

SCOM 2012 DPM 2012

Potential integration issues with System Center 2012 Virtual Machine Manager and Operations Manager

J.C. Hornbeck — Mon, 30 Jan 2012 09:47:04 GMT

Here’s a new Knowledge Base article we published today on VMM 2012 and SCOM. This one talks about a couple potential integration issues you may run into after configuring OpsMgr integration with System Center 2012 Virtual Machine Manager:

=====

Symptoms

This article presents possible errors and issues you may encounter after configuring Operations Manager integration with System Center 2012 Virtual Machine Manager (SCVMM 2012). Note that this can apply to System Center Operations Manager 2007 (OpsMgr 2007) as well as System Center 2012 Operations Manager (OpsMgr 2012).

Resolution

1. Error 11808 - Unable to set the Operations Manager root server in VMM because the Operations Manager is not installed on the Virtual Machine Manager management server.

Recommended action:

Install the Operations Console for System Center Operations Manger 2007 on the Virtual Machine manager server and then try the operation again.

NOTE This error message can be misleading as it applies to either version of OpsMgr even though it only specifically mentions OpsMgr 2007. The console required has to match the version you are using on the Operations Manager server, so if you are integrating with OpsMgr 2012 you need to install the OpsMgr 2012 console on the SCVMM 2012 server.

=====

2. Error 25907 - The credentials used to connect to Operations Managers management group do not have the necessary permissions

Recommended Action:

Try operation again with different credentials or add the credentials to the OpsMgr Administrator Role

NOTE You will need to add either one of these to the OpsMgr built-in administrators group. Add the account used for integration or add the machine name if local system was selected during integration.

=====

3. Error 10218 - Setup could not import the System Center Virtual Machine Manager Management pack into Operations Manger server because one or more required management packs are missing. The VMM management pack cannot be deployed unless the following components management packs are present in operations manager 2007…..

Recommended action:

To provide the missing component management packs, ensure that the following management packs cannot be deployed unless the following component management packs are present in operations manager 2007:…

NOTE The minimum required version of the Management Packs is 6.0.5000.0

=====

4. Error status in the operations manager settings.

Error Details:

Operations Manager discovery failed with error: "Exception of type 'Microsoft.virtualmanager.enterprisemanagement.common.dicoverydatainvalidrelationshipsourceexception0m10' was thrown.

NOTE This is caused by one or more hosts under SCVMM management not having the OpsMgr agent installed and reporting to an Operations Manager server. The new architecture in SCVMM 2012 requires that the Operations Manager agent be installed on all hosts so they can send VM information directly and not have to go through SCVMM.

=====

5. Error 25923 - PRO Diagnostics Target is not monitored

Recommended Action

Check if Operations manager agent is running on the computer running VMM management server

NOTE This is usually caused by the Operations Manager agent not being installed on the VMM server. It can also occur if the VMM server has the agent installed but it is not monitored (e.g. it there are problems with health agent). Verify the operations manager console and make sure it is in a Healthy State.

More Information

For more information see https://blogs.technet.com/b/scvmm/archive/2012/01/23/potential-issues-with-vmm-2012-opsmgr-integration.aspx

=====

For the most current version of this article please see the following:

2668088 : Potential integration issues with System Center 2012 Virtual Machine Manager and Operations Manager

J.C. Hornbeck | System Center & Security Knowledge Engineer

Get the latest System Center news on Facebook and Twitter:

How to detect and troubleshoot frequent configuration changes in Operations Manager 2007

J.C. Hornbeck — Mon, 23 Jan 2012 10:06:06 GMT

Here’s a new Knowledge Base article we published today on SCOM 2007. This one talks about configuration churn, what can cause it and how you can address it if you see it in your environment:

=====

Configuration Overview

The System Center Management Configuration service is responsible for calculating the configuration of every health service in the Operations Manager 2007 Management Group. The configuration of a health service consists of the rules, monitors, discoveries and tasks for the health service and for all the instances monitored by the health service. In order to calculate all the required configurations for each health service, the Management Configuration service needs to have a list of all instances of all monitored classes, the hosting relationships between instances, the rules, monitors, discoveries and other workflows assigned to the monitored classes, and the health services responsible for monitoring the instances. In addition, the Management Configuration service also needs to read the membership of all instance groups in the Management Group and apply any overrides for rules and monitors that have been targeted at these groups, classes or individual instances.

Objects in a management group will be defined as instances of monitored classes based on discovery data submitted by discovery workflows. If a key property of an object changes, that object may be added as a new instance of a monitored class, or no longer be considered an instance of that class. As the list of classes the object is determined to be a member of changes, the configuration for the health service that monitors that object will also change as rules, monitors, discoveries, tasks and overrides are added or removed from the previous configuration.

Configuration Churn

If a large amount of discovery data is submitted to the Management Configuration service, or submitted too fast for the Management Configuration service to process before more discovery data is submitted, agents may not be able to receive a stable configuration, as it will always be in the process of being calculated. The frequent submission of discovery data, also known as configuration churn, can cause some health services to run under old configurations, or cause the configuration of management servers to become stale, subsequently causing them to appear gray in the Operations console.

Discovery data is submitted by a health service when a discovery workflow runs. Introduction of a new Management Pack to a Management Group can cause several discovery workflows to run on each agent, and as new instances are discovered, additional discoveries may be run on some agents. Changes to groups, overrides and other workflows can cause discovery workflows to run on agents, and introduction of new agents can also cause the Management Configuration service to update the instance space with the new agent's configuration.

When a discovery workflow is configured to run too often, or the properties discovered by the workflow change each time it is run, the Configuration Management service will be forced to recalculate the health service configuration often. If this happens for many agents, or the Root Management Server (RMS) is under heavy workload already, the Configuration Management service may not be able to keep up with the rate of change and configuration churn may occur.

Identifying Configuration Churn via the RMS event log

The following event in the Operations Manager event log on the RMS indicates that the Management Group configuration has changed due to new discovery data.

Log Name: Operations Manager
Source: OpsMgr Connector
Event ID: 21024
Level: Information
Computer:
Description:
OpsMgr's configuration may be out-of-date for management group , and has requested updated configuration from the Configuration Service. The current(out-of-date) state cookie is "3A B0 1E 5C 81 F3 12 F5 56 B7 8A EF F8 01 BA 09 86 55 06 48 "

The following event indicates that the Management Configuration service has finished processing the new discovery data and calculated any changes required to the Management Group configuration based on the new data.

Log Name: Operations Manager
Source: OpsMgr Connector
Event ID: 21025
Level: Information
Computer:
Description:
OpsMgr has received new configuration for management group from the Configuration Service. The new state cookie is "34 FA 11 61 4D B8 03 59 3D 1D 66 B7 83 F3 C0 AA 7A 6F 1A 3B "

In a typical environment, every 21024 should be followed by a 21025. If the discovery data did not cause any configuration data to change, the event ID will be 21026 instead. In a large Management Group, pairs of 21024 and 21025/26 events should be expected to occur several times per hour. Long strings of 21024 events with no corresponding 21025/26 event is a sign of configuration churn. In addition, the event log may show the following event, indicating churn has been detected.

Log Name: Operations Manager
Source: OpsMgr Config Service
Event ID: 29202
Level: Warning
Computer:
Description:
OpsMgr Config Service could not retrieve a consistent state from the OpsMgr database due to too frequent database changes.
This could be due to a normal and temporary increase of discovery data; however check the most recent changes to determine if this increase is unexpected.
Most recent monitoring object change:
Instance = %1
Class = %2
Modified time = %3
Most recent monitoring relationship change:
Relationship instance = %4
Source instance = %5
Target instance = %6
RelationshipClass = %7
Modified time = %8

The Data Access Layer must read multiple tables when querying for changes. If one of the tables is modified after it is read but before all tables have been read, the Data Access Layer will log the above event and retry. If an entity or relationship instance was read during this time, information about these is included in the event fields; otherwise, these fields are left empty.

Identifying Potential Causes of Configuration Churn via the Operations Manager Datawarehouse

In management groups where the Operations Manager Reporting component has been installed, several SQL queries can be used to identify workflows that are submitting frequent changes. These queries should be run in SQL Management Studio against the Datawarehouse instance.

Total Changes Submitted by Discovery Workflows in Last 24 Hours:

select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As 'Changes'
from
(select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName,
D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName',
MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path,
ME.Name,
C.OldValue,
C.NewValue,
C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
) As #T
group by ManagedEntityTypeSystemName, DiscoverySystemName
order by count(*) DESC

This query will display three columns. The first column is the class of object at which the workflow is targeted. The second column indicates the internal name of the discovery workflow. The third column indicates the total number of property changes for all instances of this class submitted by the workflow in the last 24 hours. The total number of changes, for all classes, represents the number of times the Configuration Management service must recomputed the configuration for an agent health service.

The number of changes for some classes of objects, even in a stable environment, may not ever reach zero. Any change, such as adding or removing a property, agents being added or decommissioned, server roles being added or changed, etc. will be reflected in the numbers returned. In environments where configuration churn is experienced, one or several workflows will likely show a significantly larger value than other workflows.

Properties Changed in the Last 24 Hours:

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName,
D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName',
MET1.ManagedEntityTypeDefaultName 'TargetTypeDefaultName',
ME.Path,
ME.Name,
C.OldValue,
C.NewValue,
C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%'+MET.ManagedEntityTypeSystemName+'%'
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-24,getutcdate())
ORDER BY MP.ManagementPackSystemName, MET.ManagedEntityTypeSystemName

This query can identify which properties have changed in the last 24 hours. Combined with the previous query, this query can show what the old and new values were for the property, which agents submitted the change, the workflow that conducted the discovery, and the management pack it was contained in.

Reducing Configuration Churn

Older management packs introduced discovery workflows that submitted property changes too frequently. The current version of most management packs have modified these discovery workflows to submit data less frequently, or not query volatile properties that change frequently. Serious consideration should be given to upgrade any management pack with workflows that show up frequently in the previous query. New versions of the management pack can be downloaded from the management pack catalog: https://systemcenter.pinpoint.microsoft.com/en-US/applications/search/operations-manager-d11?q=

If a new version of the management pack is not available, or cannot be deployed at the time, the discovery interval can be adjusted via override to run less often. In some situations, the discovery responsible for the configuration churn can be entirely disabled by override. If the discovery is disabled for several weeks, the objects discovered by the workflow may be groomed out of the database, but disabling the discovery can provide a short-term solution to eliminate configuration churn before this occurs. The workflow can also be enabled for short intervals to rediscover the objects prior to them being groomed.

Some of the workflows in these older management packs are highlighted in the following blog:

https://blogs.technet.com/b/kevinholman/archive/2009/10/05/what-is-config-churn.aspx

If the workflow is from a custom discovery that targets a volatile property, such as free disk space, the discovery should be re-written to not target a property that changes often. Discovery workflows should not target instances with a short lifetime (a few weeks or less), nor collect properties of those instances that change often (more than once a month). Rules that collect performance data should be used for volatile data, as that is not considered in calculating configuration.

Additional Performance Tuning

In large management groups (> 1000 agents), the RMS may become very busy with operations that would not normally cause a problem in smaller management groups. In this situation, even a small rate of property changes could cause frequent churn, due to the length of time required to process the changes. There are a number of configuration changes that can be implemented to reduce the operational overhead of the RMS and allow it to process a normal rate of property changes quickly enough to avoid configuration churn. These configuration changes are highlighted in the following blog:

https://blogs.technet.com/b/mgoedtel/archive/2010/08/24/performance-optimizations-for-operations-manager-2007-r2.aspx

Forcing Configuration Change for the Management Group

If configuration churn for the management group is occurring constantly, any changes to reduce frequency of or disable the problem workflows will never be propagated to agents. In this case, the flow of inbound discovery data will need to be blocked to allow the System Center Configuration Management service to calculate a current configuration with the modified or disabled workflow.

Discovery data is submitted to the OperationsManager database via the System Center Data Access Service. The data is first submitted to the DAS by the System Center Management service on the RMS. The RMS gets this data from agents or other management servers. Using the Windows Firewall or some other networking means to block inbound connections to the RMS on port 5723 will prevent discovery data from being submitted to the OperationsManager database just long enough for the Configuration Management service to calculate the current configuration for the agents submitting the data.

The System Center Management service and the System Center Data Access Service on the RMS should not be stopped or disabled during this process. The System Center Configuration Management service requires a running and healthy System Center Management service on the RMS in order to complete calculation of the management group configuration. It also requires the System Center Data Access Service to communicate with the database. In addition, some data may become backlogged on the agents and other management servers during this process, so the Firewall or port exclusion should be lifted as soon as event 21025 is seen in the Operations Manager event log on the RMS, indicating that the Configuration Management service has calculated the new configuration for the management group with the disabled or modified workflows.

Identifying Potential Causes of Configuration Churn via Operations Manager Reporting

New reports were introduced with version 6.1.7599.0 of the Operations Manager 2007 R2 Management Pack. These reports provide insight into the overall volume of data being processed by the management group. These reports can be used to establish a standard baseline and to identify opportunities for tuning object discovery workflows. Once configuration churn has been identified and addressed, these reports can be used for long-term planning to prevent recurrences of churn.

The management pack can be downloaded from here: https://www.microsoft.com/download/en/details.aspx?displaylang=en&id=23081·

Data Volume by Management Pack report
The Data Volume by Management Pack report compiles information on the volume of data generated by management packs. The report lists the number of occurrences per management pack for the following data types:
- Discoveries
- Alerts
- Performance (number of instances submitted for performance counters collected by management pack)
- Events
- State changes
Data Volume by Workflow and Instance report
The Data Volume by Workflow and Instance report compiles information on the volume of data generated, broken down by workflows (discoveries, rules, monitors, etc.) as well as by instances.
There are two ways to access this report:
- In the Data Volume by Management Pack report, click one of the counts cells in the table at the top of the report to open the Data Volume by Workflow and Instance report for the management packs.
- Run the report directly from the Reporting section in the Operations console. If you run the Data Volume by Workflow and Instance report directly, you should set the parameters of the report to customize the results; this report is designed to provide details for information in the Data Volume by Management Pack report and so the default parameter settings may not provide the information you are looking for.

Query Words

Churn SCOM 2007 config

=====

For the most current version of this article please see the following:

2603913 : How to detect and troubleshoot frequent configuration changes in Operations Manager 2007

J.C. Hornbeck | System Center Knowledge Engineer

Get the latest System Center news on Facebook and Twitter:

Potential issues with VMM 2012/OpsMgr integration

J.C. Hornbeck — Mon, 23 Jan 2012 09:29:15 GMT

Hi everyone, Alvin Morales here. This article presents possible errors and issues you may encounter after setting Operations Manager integration with System Center 2012 – Virtual Machine Manager (SCVMM 2012). Note that this can apply to System Center Operations Manager 2007 (OpsMgr 2007) as well as System Center 2012 – Operations Manager (OpsMgr 2012).

1. Error 11808 - Unable to set the Operations Manager root server in VMM because the Operations Manager is not installed on the Virtual Machine Manager management server.

Recommended action:

Install the Operations Console for System Center Operations Manger 2007 on the Virtual Machine manager server and then try the operation again.

=====

2. Error 25907 - The credentials used to connect to Operations Managers management group do not have the necessary permissions

Recommended Action:

Try operation again with different credentials or add the credentials to the OpsMgr Administrator Role

=====

Recommended action:

NOTE The minimum required version of the Management Packs is 6.0.5000.0

=====

4. Error status in the operations manager settings.

Error Details:

Operations Manager discovery failed with error: "Exception of type 'Microsoft.virtualmanager.enterprisemanagement.common.dicoverydatainvalidrelationshipsourceexception0m10' was thrown.

=====

5. Error 25923 - PRO Diagnostics Target is not monitored

Recommended Action

Check if Operations manager agent is running on the computer running VMM management server

Alvin Morales | Senior Support Escalation Engineer

Get the latest System Center news on Facebook and Twitter:

OpsMgr 2007 SCOM 2007 SCOM 2012 OpsMgr 2012 VMM 2012

Custom APM Rules for Granular Alerting

J.C. Hornbeck — Mon, 23 Jan 2012 08:48:00 GMT

Michael wrote a post in August, about working with Alerts. One of the key takeaways from that post, is that – for each application and for each application component – we have FOUR Alerting rules, that can be turned on or off from the APM Template. Quoting that post:

Alerting Rules

There is a rule for each type of event we alert on: Performance, Connectivity, Security and Application Failure. We raise an individual alert when those types of events are detected in the monitored application. These alerts do not affect the health state of the monitored application since a single performance or exception event doesn’t mean your application is unhealthy.

The post above was written during Beta, and the UI has improved since then, but the rules are still there and work in the same way that was described.

In the RC build the options in the Template UI look like this:

While the Authoring Guide describes what the options are used for, here I want to show you a bit more of what settings they drive “under the hood”.

The two checkboxes on the top turn ON and OFF:

Alerting for “Performance” events (“Turn on performance event alerts”)
Alerting for “Exception” events (“Turn on exception event alerts”) – when this checkbox is enabled, it allows you to configure three more options (a breakdown of the type of exceptions):
- Security alerts
- Connectivity alerts
- Application failure alerts

Together, these checkboxes essentially enable and disable the previously-mentioned Alerting rules. You can find them in the Authoring pane of the Operations Console, under “Rules”:

The names should be all pretty self-explanatory to understand which one maps to which option.

While this mechanism is flexible enough for the most common usage, I want to show you how the whole thing works end to end, and show you how the solution is powerful and flexible, and how you can do even more with APM and configure even more granular alerts than the UI allows you to – with little XML editing.

If you look at those rules (in the Operations Console, or by un-sealing the MP and watching its XML), they are all very similar: they have a Data Source looking for the incoming APM events, and a Write Action that turns them into Alerts.

The Data Source has a configuration as follows:

As you can see in the screenshot above, the same data source is used for all four rules, and the “AspectType” is used to tell apart Performance, Connectivity, Security and Application failure events.

This is great for most situations, and our default settings have been chosen with the assumption that Operations folks would be more interested in Performance, Connectivity and Security events – those where they might be able to operate – but not necessarily about “Application failure” events, since those are (often) a bug in the code, and (typically) only a developer can fix those exceptions.

Even if this model is great, I found that, in some situations, people might want to have even more fine-grained alerting rules defined. In particular, I think the connectivity and security aspects are quite well-defined in our APM default configuration and they are typically not noisy unless something is really wrong. The same is not necessarily always true for Performance events and Application failures. For example you might want to get:

Performance event alerts only for a specific web page or method (this can also be achieved by defining a transaction, but depending on the situation one approach or the other might be preferred – I’ll explain transactions in a future post)
Performance event alerts for all cases but excluding a particular page/method which is “well known” to be slow but can’t be fixed/optimized (this is something that cannot be achieved even with a transaction)
Application failure event alerts only for certain type of exceptions and not for other ones
Application failure events alerts for all exception but excluding a particular page which is known to throw an un-handled exception but doesn’t cause bad user experience or can’t be fixed
Application failure events alerts for all exception but excluding a particular exception type which can’t be fixed by the developer
- one specific situation where #5 is desirable is when someone, calling a page which is not present on an ASP.NET application, will result in throwing a “System.Web.HttpException” with an HTTP 404 Error (not found) code – this is by design in ASP.NET: if I call an .aspx page, the ASP.NET engine will try to retrieve it and will be throwing an HTTP error; this could cause a lot of noise in case a crawler or vulnerability assessment tool hits the site searching for “well-known” but not-present pages (this is actually something that we observed on the production deployment monitoring parts of the microsoft.com website)

For all these situations (and more) there is a a fairly simple solution: writing new APM Alerting rules with an added Expression Filter. Basically we’ll have a workflow which looks like the following:

One such a sample rule is pasted below. It looks very similar to (and in fact, it is derived from) the “default” APM alerting rules described earlier – only the Condition Detection highlighted has been added. This one rule represents example #5 from the list above – essentially, it should filter out those “page does not exist” 404 errors, but still alert on every other exception.

<Rule ID="Apm.AlertAppFailureAspectRule.Exclude404.message.Sample" Enabled="false" ConfirmDelivery="true"
Target="APM!Microsoft.SystemCenter.Apm.ApplicationInstance" Remotable="false">
  <Category>AlertCategory>
  <DataSources>
    <DataSource ID="LOBProvider" TypeID="APM!Microsoft.SystemCenter.Apm.LobDataProvider">
      <Name>$Target/Property[Type="APM!Microsoft.SystemCenter.Apm.ApplicationInstanceBase"]/ApplicationName$Name>
      <AspectType>applicationfailureAspectType>
    DataSource>
  DataSources>
  <ConditionDetection ID="FilterExceptionClass" TypeID="System!System.ExpressionFilter">
    <Expression>
      <Not>
        <Expression>
          <And>
            <Expression>
              <SimpleExpression>
                <ValueExpression>
                  <XPathQuery Type="String">EventData/exceptionclassXPathQuery>
                ValueExpression>
                <Operator>EqualOperator>
                <ValueExpression>
                  <Value Type="String">System.Web.HttpExceptionValue>
                ValueExpression>
              SimpleExpression>
            Expression>
            <Expression>
              <RegExExpression>
                <ValueExpression>
                  <XPathQuery Type="String">EventData/messageXPathQuery>
                ValueExpression>
                <Operator>ContainsSubstringOperator>
                <Pattern>does not existPattern>
              RegExExpression>
            Expression>
          And>
        Expression>
      Not>
    Expression>
  ConditionDetection>
  <WriteActions>
    <WriteAction ID="AlertWriteAction" TypeID="Health!System.Health.GenerateAlert">
      <Priority>1Priority>
      <Severity>1Severity>
      <AlertMessageId>$MPElement[Name='Apm.AlertAppFailureAspectRule.Exclude404.message.Sample.AlertMessage']$AlertMessageId>
      <AlertParameters>
        <AlertParameter1>$Target/Property[Type="APM!Microsoft.SystemCenter.Apm.ApplicationInstanceBase"]/ApplicationName$AlertParameter1>
        <AlertParameter2>$Target/Host/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/PrincipalName$AlertParameter2>
        <AlertParameter3>$Data/EventData/exceptionclass$AlertParameter3>
        <AlertParameter4>$Data/EventData/message$AlertParameter4>
        <AlertParameter5>$Data/EventData/name$AlertParameter5>
        <AlertParameter6>$Data/EventData/ViewDetail$AlertParameter6>
      AlertParameters>
      <Suppression>
        <SuppressionValue>$Data/EventData/eventConsolidationHash$SuppressionValue>
      Suppression>
      <Custom1>ApplicationFailureCustom1>
    WriteAction>
  WriteActions>
Rule>

This rule still produces alerts for other exceptions that “look and feel” pretty much like the built-in rules, but will not raise an alert for those HTTP 404’s “file does not exist” errors. Although be aware that the example above will not work on localized .NET Framework/Windows version, because I am searching for a English string (“does not exist”) in the error message. This is not really meant as a production-quality MP considering all cases, just as a quick example of how you can build your own workflows by adding filtering criteria, and my goal it mostly to help you understand how the APM pieces fit together in Operations Manager 2012 so that, with that knowledge, you can get creative and adapt it to your needs.

Anyway, an alternative would be digging out the actual HTTP Error code, which is buried down in the DataItem as well. To do so, we can rewrite our Expression Filter as follows:

<ConditionDetection ID="FilterExceptionClass" TypeID="System!System.ExpressionFilter">
  <Expression>
    <Not>
      <Expression>
        <And>
          <Expression>
            <SimpleExpression>
              <ValueExpression>
                <XPathQuery Type="String">EventData/exceptionclassXPathQuery>
              ValueExpression>
              <Operator>EqualOperator>
              <ValueExpression>
                <Value Type="String">System.Web.HttpExceptionValue>
              ValueExpression>
            SimpleExpression>
          Expression>
          <Expression>
            <SimpleExpression>
              <ValueExpression>
                <XPathQuery Type="String">EventData/log/events/event[1]/variable[1]/variables/variable[name='_httpCode']/valueXPathQuery>
              ValueExpression>
              <Operator>EqualOperator>
              <ValueExpression>
                <Value Type="String">404Value>
              ValueExpression>
            SimpleExpression>
          Expression>
        And>
      Expression>
    Not>
  Expression>
ConditionDetection>

Since the code above is is not easily readable due to the blog layout (but should be possible to copy/paste it just fine), a Management Pack with both variations of this rule is attached at the end of this post. It also contains two more (fairly similar) examples, for a total of three rules based on “Application Failure” events and one on “Performance” events. All the rules are disabled by default to prevent duplicate alerts starting to appear in your environment as soon as you import the MP – if you use these rules, you might want to disable the checkboxes in the template for the “built-in” rules, first. You can then decide to turn these new rules on by default, or selectively thru overrides, as you would do with any other rule.

Please also note that some of the criteria contained in the condition detection filters cannot be edited from the Operations Console, and alert messages token replacement will also most likely break if edited thru the GUI. These things are best edited in XML.

In addition, these criteria might need to be revisited fairly often as part of your tuning – known problems could became worth considering again, and new issue might appear that you want to start filtering out, and so on.

These rules with their filters will only affect Alerting; all APM events will still be collected in the database and be visible thru the AppDiagnostics console – as the actual event insertion is driven by a different rule (using the same data source module, but a different write action).

Once events are stored and visible in AppDiagnostics, we also provide ways to automatically delete them from the database, or mark as “by design” those events that aren’t considered useful or interesting, or appear to add noise. This is the Problem Management feature in AppDiagnostics (Rules Management Wizard), which – while it doesn’t prevent events from being stored or alerts from being raised in the first place - it helps keeping your database “clean” and I like to consider it a sort of “intelligent” grooming. There would be a lot to be said about the Problem Management feature - I’ll try to come back to this feature and its rules in a future post.

Happy .Net Monitoring!

Disclaimer

This posting is provided "AS IS" with no warranties, and confers no rights. Use of included utilities are subject to the terms specified at https://www.microsoft.com/info/copyright.htm.

Microsoft Enterprise Product Roadmap Webcast

J.C. Hornbeck — Thu, 19 Jan 2012 09:18:15 GMT

Please join us for a high level discussion into Microsoft software investments designed to cover most of the major product lines. This will be a 35,000 foot view into the following areas:

Windows Family
Virtualization
Office System
Unified Communications
IT Management
Security

This particular webcast is scheduled for Friday, January 20 at 1:00pm Pacific Time although others are scheduled as well. See this link for more information.

J.C. Hornbeck | System Center & Security Knowledge Engineer

Get the latest System Center news on Facebook and Twitter: