Operations Manager 2007 Service Pack 1 Debug Tracing
As Marius discussed (https://blogs.msdn.com/mariussutara/archive/2008/01/21/operations-manager-2007-sp1-tracing.aspx) SP1 introduces some changes in SP1 to debug level tracing sometimes required when working with Customer Support Services (CSS) on issues "under the hood". The KB article discussing tracing in the original release product (https://support.microsoft.com/kb/942864) will undergo a change shortly to reflect these changes.
I would still not expect tracing to be used without CSS interaction simply because of the detailed nature of the trace output which in MANY cases isn't meaningful without access to source code and the developers that wrote it:)
Below are my raw (warts and all) recommendations for the KB article update (should be prettier in the final KB release):
INTRODUCTION
This article describes how to use diagnostic tracing in Microsoft System Center Operations Manager 2007 and in Microsoft System Center Essentials 2007.
Important: We recommend that you perform diagnostic tracing only in association with a Microsoft Customer Support Services (CSS) representative. In Operations Manager and Essentials original release products we recommend this because the generated traces contain no human-readable text. After the traces are converted by a Microsoft CSS representative, human-readable text is available. However, this text contains only low-level information such as source-code file names, locations, source-code functions, and return codes. This information may be helpful if you have to troubleshoot a complex issue.
Within the Service Pack 1 release and above, Trace Message Files (TMF) are supplied which allow conversion of binary trace files to text. However it is still strongly recommended diagnostic tracing only be performed when required, and with the consultation of a Microsoft representative.
MORE INFORMATION
System Center Operations Manager 2007 and System Center Essentials 2007 implement a diagnostic tracing method that differs from earlier versions of Microsoft Operations Manager. This new tracing method creates binary files in which to store tracing information. Because this new tracing method is implemented at the Windows kernel level, it is highly efficient, and it can log tens of thousands of trace messages per second.
Trace information contains information about the context of a text-based trace message. However, within the original release products this trace information is not in a human-readable format. The traces must be sent to Microsoft CSS for conversion. With the Service pack 1 releases of Operations Manager and Essentials this is no longer a requirement as the files required to convert traces to human-readable format are supplied.
This article discusses the tools that are available to start and to stop tracing if you are asked to do this by a Microsoft CSS representative together with new functionality provided within Service Pack 1 and above.
Tracing tools location
The installation folder for each role, such as the Agent role, the Management Server role, or the Gateway role, contains a folder that is named Tools. The following files are located in the Tools folder:
StartTracing.cmd
StopTracing.cmd
TracelogSM.exe
TracingGuidsBid.txt
TracingGuidsUI.txt
TracingGuidsNative.txt
TracingReadMe.txt
Service Pack 1 introduces the following additions:
FormatTracing.cmd
ViewRealtimeTracing.cmd
TraceFmtSM.EXE
OpsMgrTraceTMF.cab
Default.TMF
System.TMF
To start tracing
NOTE: Within Service Pack 1 and above tracing is automatically started on both agent and management server roles. Tracing will default to error output only with binary trace files being written to the %WINDIR%\Temp\OpsMgrTrace folder. Before the tracing level for trace output can be changed, tracing must first be stopped.
To start diagnostic tracing, follow these steps:
1. On the computer on which you want to start tracing, click Start, click Run, type cmd , and then click OK.
2. At the command prompt, use the cd command to change to the Tools directory. For example, type the following command, and then press ENTER:
cd\Program Files\System Center Operations Manager 2007\Tools
3. Type StartTracing LEVEL , and then press ENTER. In this command, replace LEVEL with the tracing level that you want. You must use uppercase characters to specify the tracing level. The following levels are available :
ERR
WRN
INF
VER
For example, type StartTracing WRN .
When you start tracing, the trace output is written to the following binary files:
MOMTraceNative.etl
MOMTraceBID.etl
MOMTraceUI.etl
These files are located in the %WINDIR%\Temp folder.
NOTE: In Service Pack 1 and above trace files (.etl) are located in the %WINDIR%\Temp\OpsMgrTrace folder.
To stop tracing
To stop diagnostic tracing, follow these steps:
1. Click Start, click Run, type cmd , and then click OK.
2. At the command prompt, use the cd command to change to the Tools directory. For example, type the following command, and then press ENTER:
cd\Program Files\System Center Operations Manager 2007\Tools
3. Type StopTracing.cmd , and then press ENTER.
After you stop diagnostic tracing, you can send the .etl files from the %WINDIR%\Temp folder to the Microsoft CSS representative with whom you are working.
NOTE: In Service Pack 1 and above trace files (.etl) are located in the %WINDIR%\Temp\OpsMgrTrace folder.
To Convert Existing Traces to Text Format (Available with Service Pack 1 and above only)
To convert existing trace files to text tracing should first be stopped via the StopTracing,cmd command. Information within the converted traces is of an extremely low level detail (debug level) and will not be of use to the
1. Click Start, click Run, type cmd , and then click OK.
2. At the command prompt, use the cd command to change to the Tools directory. For example, type the following command, and then press ENTER:
3. cd\Program Files\System Center Operations Manager 2007\Tools
4. Type FormatTracing.cmd , and then press ENTER.
The first time FormatTracing.cmd is executed the Trace Message File (TMF) required to convert traces is extracted to the TMF subfolder.
The FormatTracing.cmd batch file then enumerates each trace file (*.etl) in the %WINDIR%\Temp\OpsMgrTrace folder, executing the TraceFMTSM.exe utility to convert it to text. The text output will be written to a file of the same name with the extension .log in the %WINDIR%\Temp\OpsMgrTrace folder. A summary (.sum) file will also be written to the same location for each file converted detailing each trace message event converted.
Viewing Converted Trace Files (Available with Service Pack 1 and above only)
Trace files converted to text (%WINDIR%\Temp\OpsMgrTrace) using the FormatTracing.cmd batch file can be viewed with Notepad or another suitable text editor.
Boot Time Tracing is Enabled by Default (Available with Service Pack 1 and above only)
On a typical Root Management Server (RMS) installation three trace sessions will begin when the HealthService service is started. By default only error tracing is performed, which will typically write little or no information to the trace files. The default trace file location and names created are:
%WINDIR%\Temp\OpsMgrTrace TracingGuidsBID.etl - Trace output for Managed Code Components
%WINDIR%\Temp\OpsMgrTrace TracingGuidsNative.etl - Trace output for Native Code Components
%WINDIR%\Temp\OpsMgrTrace TracingGuidsUI.etl - Trace output for Managed code User Interface (OpsMgr UI)
Each session is enabled with circular tracing at a maximum files size of 100Mb. The typical etl file size for a newly created etl file is 16Kb.
NOTE: Although an Operations Manager Role such as an Agent role does not contain Managed or UI tracing messages, the three default trace ETL files will be created by default when the HealthService service is started. On a User Interface only role no trace sessions are started by default.
How to disable boot time tracing (Available with Service Pack 1 and above only)
Three trace providers are automatically started on any role which contains the HealthService service with their trace files written to %WINDIR%\Temp\OpsmgrTrace. Typically the etl files in this folder will be small, especially on the agent role. However on the RMS role, if the operations manager installation becomes unhealthy one or more of these files could potentially grow to their maximum configured size of 100Mb each (TracingGuidsBid.etl, TracingGuidsNative.etl & TracingGuidsUI.etl, together with Previous.<filenames>.etl) even when error tracing only is enabled.
If it becomes necessary to disable tracing, due to small boot partition size this can be accomplished by adding a registry value on the computer. Note: If tracing is disabled this may affect future troubleshooting of Operations Manager because any low level tracing information will be lost. Future hotfixes, service packs or product updates may remove the registry entry therefore enabling tracing once more.
To disable debug level tracing add the following registry key to the Management Server or Agent you wish to disable tracing on:
HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Tracing\DisableAutoTracing:REG_DWORD:0x1
This registry key will not exist be default. Removing the registry key or changing its value to 0 will enable default tracing.
If you execute the StopTracing.cmd batch file from the Tools folder the Operations Manager trace sessions will be stopped. The etl files in the %WINDIR%\Temp\OpsmgrTrace can be deleted if they are no longer required to regain disk space.
How to move the default location of trace file output (Applies to Service Pack 1 and above only)
If, due to boot partition space limitations (typically the Management Server Role only) it is necessary to move the default location of trace files (*.etl) the following procedure can be used:
· Create a new folder on a local disk with sufficient capacity to house the files (three default trace files at maximum 100Mb, three previously used trace files at maximum 100Mb = Total potential trace file capacity of 600Mb).
· Set at least the following NTFS permissions on the folder, SYSTEM = Full Control, Administrators = Full Control.
· Locate the StartTracing.cmd batch file in the Tools folder beneath the Operations Manager/Essentials installation folder and opening it in a text editor such as Notepad.
· Within the StartTracing.cmd batch file locate the SET OpsMgrTracePath statement and replace the default value %windir%\temp\OpsMgrTrace with the path to the new location (delimit the path with speechmarks (“) if the path contains a space character)
· Save the changes to StartTracing.cmd
· To begin tracing to the new folder execute the StopTracing.cmd batch file and then execute the StartTracing.cmd batch file. Shortly after executing the StartTracing.cmd batch file the TracingGuidsBid.etl, TracingGuidsNative.etl & TracingGuidsUI.etl files will be written to the new folder.
· Before using the FormatTracing.cmd command to convert traces to text, this file much be edited too to change the OpsMgrTracePath variable to point to the new trace file folder.
Future hotfixes, service packs or product updates may change the files in the Tools folder changing the functionality back to the defaults. It is advised that you check the changes remain in place after performing updates.
How to view real time tracing (Available with Service Pack 1 and above only)
The use of the ViewRealTimeTracing.cmd batch file to view real time traces is strongly discouraged in a production environment without advice of Microsoft Customer Support Services. Viewing traces in real time especially at Information or Verbose trace level settings can incur large performance penalties.
The first time ViewRealTimeTracing.cmd is executed the Trace Message File (TMF) required to convert traces is extracted to the TMF subfolder. The ViewRealTimeTracing.cmd batch file then enumerates each trace session definition file (Tools\TracingGuids*.txt) in the Tools folder, launching a separate instance of TraceFmtSM within a command window. Output of trace sessions for Managed, Native and UI Managed traces is output to each window.
The ViewRealTimeTracing.cmd batch file accepts a single, optional command line parameter (-ods). Supplying this parameter will additionally output trace data to a debugger.