Default SCOM Windows agent tracing?

Question

Today we found out something I've never seen before but might already be there for a long time and seems to happen on all windows agents.

A windows agent (2019ur3) always creates trace files in C:\Windows\Logs\OpsMgrTrace upon starting which may grow till about 100MB per file.
Running the stoptracing.cmd command doesn't resolve this, the files just start over again on agent startup.

Probably caused by a few txt files in C:\Program Files\Microsoft Monitoring Agent\Agent\Tools
Like this one: TracingGuidsAPM.txt
If i remove the file and restart this seems to stop that tracing.

Further in the eventlog "Microsoft-Windows-Kernel-EventTracing/Admin" i see errors like this that relate to the tracing:
Error
EventID 2
Session "TracingGuidsApmConnector" failed to start with the following error: 0xC0000035

Now my questions are

1. Is it normal that these traces run out of the box and should i keep them enabled and why?
2. Do i need to take action on that error event and what may that be?

Answer 1

Hi Ronald (@Berg, Ronald van den ),

here are my comments to your questions:

Is it normal that these traces run out of the box and should i keep them enabled and why?

Yes, this is by default (tracing is runnning). The SCOM ETL Tracing is enabled by defaultl, but runs on a lower logging level. This also why, if you want to do a VERBOSE tracing, you first need to stop the current (default) one (stoptracing.cmd) and enable the VERBOSE after that (starttarcing.cmd VER).

Leaving the trace running does not seem to be an issue as the file that is written is circular and gets overwritten. If you stop ti, suing "stoptracing.cmd" it will start again on the next service rstart. Still, if you decide to stop it, here is how you do it:

Use diagnostic tracing in System Center Operations Manager and in System Center Essentials

https://learn.microsoft.com/en-us/troubleshoot/system-center/scom/use-diagnostic-tracing

Do i need to take action on that error event and what may that be?

ETL tracing is done based on the so called Tracing GUIDs, which uniquely identify the components that needs to be traced. In this case the event states that a certin Tace provider session could not be started. The error resolves to "STATUS_OBJECT_NAME_COLLISION". If I am no mistaken you can stop those events, by doing some registry key adjustments to:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\Autologger\EventLog-System{b675ec37-bdb6-4648-bc92-f3fdc74d3ca2}

Please check those:

Fix “Error Code: 0XC0000035” Kernel Event Tracing on Windows?

https://appuals.com/kernel-event-tracing-error-0xc0000035-windows/

and

Session "PerfDiag Logger" failed to start error: 0xC0000035 Event ID 2, any clues?!

https://social.technet.microsoft.com/Forums/office/en-US/f505d547-4f95-4e96-83a2-c31f33139e53/session-quotperfdiag-loggerquot-failed-to-start-error-0xc0000035-event-id-2-any-clues?forum=win10itprogeneral

Here are a couple of references to support the facts:

Tracing SCOM Workflows with PowerShell

https://monitoringguys.com/2020/12/15/tracing-scom-workflows-with-powershell/

Use diagnostic tracing in System Center Operations Manager and in System Center Essentials

https://learn.microsoft.com/en-us/troubleshoot/system-center/scom/use-diagnostic-tracing

How to collect and analyze a SCOM (System Center Operation Manager) ETL Trace in depth. Version Independent

http://www.kuskaya.info/2019/05/01/how-to-collect-and-analyze-a-scom-system-center-operation-manager-etl-trace-in-depth-version-independent/

I hope I could help out.

---

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)
Regards
Stoyan Chalakov