The Rule "Memory Available Megabytes 2008" from the Windows server 2008 Operating System (monitoring) management pack collects data for the Performance object: memory and performance Counter "Available Mbytes" every five minutes.

It uses Optimization with tolerance value set to 10 and Maximum Sample separation value set to 12.

The issue that we have with this rule is that it collects and writes the Sample data into DW from the server even when the Sample value did not change between the last sampled and the current sampled value.

From the output:

Server1 Memory Available MBytes 234 8/6/10 17:08

Server1 Memory Available MBytes 234 8/6/10 17:03

There are many such instances in the output where two consecutive samples have the same value but still they are written to the DW.　

Solution:

Anytime the rule / datasource submits data to the DB, it posts both the last posted value as well as the new value.

Example: If the maximum sample separation is 12, the tolerance is 5, and the frequency is 60s. let us take this case where the raw data collected reads like this: 10, 11,11, 11,11, 11,11, 11,11, 11,11, 11,11.

All the 11s will be omitted because of the tolerance being 5. The last 11 is submitted because it is time for a periodic sample. Additionally, another data value is posted with value 10 with time 60s before the last 11, so that the UI will show a flat line at 10 for the whole period, followed by step increase to 11.

In short, the behavior is expected. The reason for the additional values is to enable the UI to draw a step graph rather than join 10 (at time 0) to 11 (12 samples later) using a straight line. This could be misleading if the value changed from 11 on 11th sample to 10000 on the 12th sample, because we would then draw a straight line from 10 to 10000, which would cause the user to think that the increase was gradual and that the data for the intermediate points in time was higher than the actual value (which is 11).

There are several sources of troubleshooting information for determining the cause of a failed Windows Computer agent install from the Operations Manager UI.

1.       Task output from the install.  The error message(s) are displayed in the task output.  Click on the target in the top panel to see the corresponding success or error message.  Agent installs are console tasks.  You can also see the messages under Monitoring \ Task Status in the Operations Manager UI.

2.      Event log. Agent installs and uninstalls are logged to the Operations Manager event log, under the source of Health Service Modules. (Most common event IDs: 10602, 10616).

3.       Agent install logs.  Depending on how far the install process made it before failing, the agent push install logs are copied to Operations Manager 2007 install folder \ AgentManagement\AgentLogs folder on the Management Server you selected under the “Select Objects to Manage” panel in the discovery wizard.  If the agent you are trying to install is not available to the Management Server, none of the logs would have been generated and you will need to review the tasks status and the event log on the management server.

There are 3 types of agent install logs:

a.       <Agent NetBios name>AgentInstall.log – this is the MSI log from the agent.

b.      <AgentNetBios name>AgentMgmt.log  - this is the bootstrapper log (bootstrapper launches the MSI with the correct switches).

c.       <AgentNetBios name>AgentPatch.log – this contains MSI logging for OpsMgr patches, such as OOMADS (for Active Directory).

4.       ETL logs. Agent push is performed through the health service on the management server you have selected.  You can use the TracingGuidsNative.etl log to further troubleshoot a failed install.  Keep in mind that (1) this contains all of the native code health service activity (you will need to filter the log to find what you are looking for and (2) by default the trace level is ERR.  I would recommend using a trace level of INF.  A better approach may be to use traceconfig.exe in the Ops Manager install folder \ tools folder.  There is an entry specifically for Push Agent Module that if selected will produce an etl trace file with only agent push messages.

Here’s how to get the trace information:

1.       Before starting the agent push installs, open TraceConfig.exe from the Ops Manager install folder \ tools folder.

2.       Select the “Push Agent Module” trace provider.

3.       Select Tracel Level of Information.

4.       Start the trace.

5.       When the agent push completes, stop the trace.

6.       From a command prompt, path to the Ops Manager install folder \ tools folder.

7.       Run formattracing.cmd

8.       When formattracing.cmd completes, it will popup Windows Explorer at the %windir%\temp\OpsMgrTrace folder.  The formatted  traceconfig output is in the OpsMgrCustom.log. 

As of today, we are seeing this caused by a file called ~0000.exe with a time stamped year of 1601.  This is a file used by Symantec's Ghost product.

The path to this file may be:

There are 2 known workarounds:

1.  Delete the file on the client and run a full inventory.

2.  Exclude the folder that this file lives in from being scanned and run a full inventory.

The sinvproc.log will indicate the client its processing when the error occurs:

...

Make sure these 2 entries reference the same thread Id (the hex number at the end of the line, as sinvproc can by multithreaded):

WARNING: Exception Detected: Unspecified error~~ !! SMS_SOFTWARE_INVENTORY_PROCESSOR 1/9/2008 11:45:52 AM 13768 (0x35C8)

As of today, the ~0000.exe with a year time stamp of 1601 is the only know file that is causing this error.  I will update the blog if I hear of any others. 

Update (5/19/2008)

From Symantec 

https://service1.symantec.com/SUPPORT/on-technology.nsf/docid/2008031311003660

Hotfix for SCCM:

https://support.microsoft.com/kb/950653/en-us

Sorry for the delay...

Before we look at specific MOM logs and what information they contain, let's talk about the types of logging available.

In part 1, we talked about the .mc8 logs and the registry keys that are used to tweak logging.  There are a couple of other registry keys that we need to talk about.

Das Logging (Database Access Services) - this logging (under dllhost*.mc8 log files), records the DAS activity - data into the Onepoint database.

Its set using the following key:

HKLM \ Software \ Mission Critical Software \ DASServer \ LoggingFlags set to ffff. (0 is disabled)

Tracelevel (covered in Part 1) must be set to 9 for Das Logging to be enabled.

To activate it, you have to restart the MOM Das Com+ Application.  These logs are written to the profile of the DAS account, under Local Settings \ Temp \ Microsoft Operations Manager. 

The Das log is useful for troubleshooting SQL type errors, or data stream type issues, like seeing the performance counter data that is being inserted into Onepoint.

Das logging should not be enabled unless you are troubleshooting a database related issue - keeping Das Logging enabled can cause performance degradation in writing to the Onepoint database.

A second type of logging is dbconnector logging.  Dbconnector logging is described in 901049:

https://support.microsoft.com/?kbid=901049

Basically, enabling this will cause all data items (events, performance data, alerts, and discovery data) that are inserted into the Onepoint database, to also be written to dbconnector.xml, under %windir% \Temp\Microsoft Operations Manager.

This file will grow very large very quickly, so do not leave this logging enabled unless you are trying to capture specific data from your agents.

Finally, DebugView from Sysinternals is a great utility for getting additional logging from MOM, especially during setup (in addition to the verbose MSI logs), and the MMPC (MOM to MOM Product Connector) execution.  In fact, DebugView provides verbose DAS type logging for the MMPC, normal DAS logging does not capture any additional MMPC database activity.

In the 3rd part (which will be out much sooner that part 2), we will go through the other MOM 2005 logs and what they are used for.

Logging is set in the registry on both the management servers and clients in this key:

HKLM \ Software \ Mission Critical Software \

Values of note:

Tracelevel - this is the verbosity level of logging.  Values are ffffffff (disabled), and 0 thru 9.  The higher the number, the more verbose the logging.  By default, tracelevel is set to 1.  When troubleshooting an issue, tracelevel 6 is typically verbose enough to indicate an error condition.  Tracelevel should only be set to 9 when debugging an issue; do not leave the value at 9; you could see a performance hit with MOM. 

Tracecircularlines - this is the number of lines that are stored in a MOM log file.  When this limit is reached, the current log is renamed and a new current log is started. The naming convention will be discussed shortly.

Traceinitseconds - this is the amount of time that will elapse when the service is started when the log file is renamed to an (init) log.  There is no line limit for the (init) log, strictly a time limit.  The default is 90 seconds.

Naming convention

Let's start with the basic .mc8 MOM logs.

Initially, the MOM Service log is called momservice.mc8 (stored in %windir%\temp\Microsoft Operations Manager\ by default).  When the traceinitseconds value elapses (90 seconds by default), the momservice.mc8 is renamed to momservice(init).mc8.  The current log then becomes momservice(b).mc8.  After this, when the tracecircularlines value is met in the momservice(b).mc8, it is renamed to momservice(a).mc8 and the new current log is momservice(b).mc8.

In addition, when you start the MOM service, the existing momservice(init).mc8, momservice(b).mc8 and the momservice(a).mc8 are appended with a 1 to become momservice(init)1.mc8, momservice(b)1.mc8 and momservice(a)1.mc8.  MOM will keep 4 copies of these (the 4 last runs of the MOM service - i.e. momservice(init)4.mc8).

The momservice*.mc8 logs should not be confused with the momservice.log (different extension) - the momservice.log contains computer discovery logging where as the momservice*.mc8 logs contain such things as agent communications, rule processing, notification activities, etc.).

The next post will list the different MOM 2005 logs and what they contain.

If you enable logging (tracelevel = 6), the momservice(b).log (on the management server) will contain the error:

Wrn:Failed to disconnect remote connection\\xx.xx.xx.xxx , error text = 2250:This network connection does not exist.

This looks a lot like the issue described in 912998 (https://support.microsoft.com/kb/912998/en-us) where the Network access: Do not allow storage of credentials or .NET Passports for network authentication  group policy is enabled.  Make sure this is not enabled on the Management Server.  If it's not enabled, most likely your Management Server action account is set to Local System.

When you installed MOM initially, you had to specify an account for the Management server action account.  You can set the Management Server action account to Local System by setting:

HKEY_LOCAL_MACHINE\SOFTWARE\Mission Critical Software\OnePoint\Configurations\ <configuration_group_name>\AA\ActionIdentityMode to 0

This is documented in 891602 (https://support.microsoft.com/kb/891602/en-us).

Restart the MOM Service for this change to take effect.

You can confirm what the Management Server action account is currenly set to using the setactionaccount.exe utility that is installed in the MOM folder (Program Files\Microsoft Operations Manager 2005 by default).

This command will give you the account:

setactionaccount <configuration_group_name> -query

If the Management Server action account is Local System, the response will be:

"Providers and responses run under the services process identity"

To change the Management Server action account back to a domain account, you need to do the following:

1. Set  HKEY_LOCAL_MACHINE\SOFTWARE\Mission Critical Software\OnePoint\Configurations\ <configuration_group_name>\AA\ActionIdentityMode to 1

2. Run the setactionaccount.exe:

setactionaccount.exe <configuration_group_name> -set <domain> <password>

The utility will prompt you for the password twice. Once the action account is set back to a domain account, you will be able to upgrade, install, uninstall, and update agent settings specifying an account (other than the action account) with local administrator rights on the target agent.