Event Tracing for Windows and Network Monitor
Event Tracing for Windows, (ETW), has been around for quite a while now as it was introduced in Windows 2000. It's basically instrumented logging that describes what a component is doing. Conceptually, it’s something like the proverbial printf("here1") concept used by programmers, but it is present in retail builds . When you enable logging in a component the result is an ETL (Event Trace Log) file. What’s new is that that Network Monitor can read any ETL file. And with the supplied parsers many network oriented ETW providers can be decoded.
How ETW Works
The idea was to standardize tracing so that it could be turned on for any component with a consistent interface. Before ETW, it was common practice to place DEBUG statements that would output to the debugger. But this often required checked builds of the binaries, special registry entries, and super secret knowledge that sometimes required code access. Now the consumer/provider model with built in enumeration advances and standardizes logging.
Unfortunately, it's not all perfect. As the OS has evolved, so has the story for collecting the ETW data. But the great news is that in Windows 7, you can collect data by scenarios and even include the raw network traffic data, all using ETW tracing.
Subscribing to a Provider
ETW uses a subscription model. Subscribing tools provide a GUID or provider name to receive the logging from that component. Network Monitor is not a subscriber at this point, so we'll have to use other tools to gather the ETW data.
Various operating systems contain different ways of getting ETW traces, but one that seems to work on most Windows operating systems is Logman.exe (https://technet.microsoft.com/en-us/library/bb490956.aspx). When you run Logman, you supply a GUID which identifies a particular provider. To list the providers you can use the /query command.
]]logman query providers
Provider GUID
-------------------------------------------------------------------------------
.NET Common Language Runtime {E13C0D23-CCBC-4E12-931B-D9CC2EEE27E4}
ACPI Driver Trace Provider {DAB01D4D-2D48-477D-B1C3-DAAD0CE6F06B}
Active Directory Domain Services: SAM {8E598056-8993-11D2-819E-0000F875A064}
Active Directory: Kerberos Client {BBA3ADD2-C229-4CDB-AE2B-57EB6966B0C4}
Active Directory: NetLogon {F33959B4-DBEC-11D2-895B-00C04F79AB69}
...
Microsoft-Windows-Winsock-AFD {E53C6823-7BB8-44BB-90DC-3F86090D48A6}
Microsoft-Windows-Winsock-WS2HELP {D5C25F9A-4D47-493E-9184-40DD397A004D}
...
see many providers listed, but we only provide ETW parsers for a subset of these. This is mainly because our focus for this feature is Windows 7 network troubleshooting. As we extend the parser set on https://www.codeplex.com/nmparsers, we will add new support for providers. In fact as we speak we are looking to publish an ETW parser for USB 2.0.
Capturing an ETW Trace with Logman
To capture ETW data, use Logman along with the GUID or provider name and some other parameters. Winsock is the layer that applications use to communicate over TCP. Let's capture all Winsock related ETW events.
logman start my_winsock_log -p Microsoft-Windows-Winsock-AFD -ets
We send it the start command and the name of our session and log file my_winsock_log. We pass it the name of the provider with -p. Alternatively, we could use the {GUID} instead. Finally we pass "-ets" which says to start logging now.
logman stop my_winsock_log -ets
This command will stop the logging, again based on the name you specified when you started. When this command completes you should have a my_winsock_log.etl file that we can be opened with Network Monitor 3.3 or 3.2.
Windows 7 Scenario based ETW Tracing
The story for ETW becomes much more interesting in the Windows 7 and Server2008 R2 world. While Logman still works, Netsh becomes the tool of choice for collecting ETW traces. Also incredibly useful is the addition of tracing by scenario and the ability to capture network traffic via NDIS with ETW tracing. But more on this a little later.
Netsh to Collect an ETW Trace.
Netsh used to be component centric with regards to tracing. But now tracing is a top level command that you can use to start and stop tracing. So here's an example that works like our previous Logman example.
]] netsh trace start provider=Microsoft-Windows-Winsock-AFD TraceFile=my_winsock_log2.etl
Trace configuration:
-------------------------------------------------------------------
Status: Running
Trace File: my_winsock_log2
Append: Off
Circular: On
Max Size: 250 MB
Report: Off
]] netsh trace stop
Correlating traces ... done
Generating data collection ... done
...
Netsh and Scenarios
Scenarios are things like "InternetClient" or "AddressAcquisition". A tracing scenario is defined as a collection of selected event providers. Providers are the individual components of the network protocol stack, such as WinSock, TCP/IP, Windows Filtering Platform and Firewall, Wireless LAN Services, or NDIS.
One of the primary goals for improved network tracing is to allow collection of all relevant information in one step, and then easily view all events associated with a specific action across the network stack. Network tracing provides a quick method for collecting information and diagnosing networking issues by logging events from all providers in the scenario, and then correlating these events by activity. In other words, related events & network packets are grouped together for given activity across different components in the networking stack, from Winsock down to NDIS.
Let's look at a list of possibilities for Windows 7.
]] netsh trace show scenarios
Available scenarios (18):
-------------------------------------------------------------------
AddressAcquisition : Troubleshoot address acquisition-related issues
DirectAccess : Troubleshoot DirectAccess related issues
FileSharing : Troubleshoot common file and printer sharing problems
InternetClient : Diagnose web connectivity issues
InternetServer : Troubleshoot server-side web connectivity issues
L2SEC : Troubleshoot layer 2 authentication related issues
LAN : Troubleshoot wired LAN related issues
Layer2 : Troubleshoot layer 2 connectivity related issues
MBN : Troubleshoot mobile broadband related issues
NDIS : Troubleshoot network adapter related issues
NetConnection : Troubleshoot issues with network connections
P2P-Grouping : Troubleshoot Peer-to-Peer Grouping related issues
P2P-PNRP : Troubleshoot Peer Name Resolution Protocol (PNRP) related issues
RemoteAssistance : Troubleshoot Windows Remote Assistance related issues
RPC : Troubleshoot issues related to RPC framework
WCN : Troubleshoot Windows Connect Now related issues
WFP-IPsec : Troubleshoot Windows Filtering Platform and IPsec related issues
WLAN : Troubleshoot wireless LAN related issues
To enable a scenario and stop it you type the following commands:
netsh trace start scenario=InternetClient
netsh trace stop
Again, an ETW trace file is created but now there are multiple providers within the same trace file. This can help you correlate tracing from multiple places as the problem moves from one component to another.
Icing is "capture=yes"
One of the coolest parts of this new tracing in Windows 7 is that you can capture NDIS network traffic using Netsh. By enabling the capture parameter, you capture network traffic. Not only is this useful for correlating to other component events, but it provides another way to get a network capture.
Since Netsh is remote-able, you could also use this to start a capture on another machine! And using the persistent=yes parameter, you can enable logging during boot to troubleshoot those nagging start up issues.
By adding the capture=yes parameter to any scenario or specific provider collection, network traffic is captured, as well. To get a simple trace, use this command:
netsh trace start capture=yes
The resulting trace contains TCP, IPv4, Ethernet, ...etc. just like a trace taken from the network interface. However the data has extra headers on the front. If you have other provider information mixed in, and the latest parsers form https://www.CodePlex.com/nmparsers, a simple filter to show you just the network traffic is as follows:
NDISPacCap_MicrosoftWindowsNDISPacketCapture
We also update the standard filters on https://www.codeplex.com/nmparsers. As new requests and ideas for standard filters come up, we'll add to the current set.
Netsh Reference
Here's a small table to summarize the Netsh commands referenced:
Start |
Starts a trace session |
Stop |
Stops a trace session |
Capture=yes |
Turns on raw network capture from NDIS |
Provider=ProviderName |
Enables tracing for a specific provider |
Show Providers |
Shows a list of providers |
Scenario=ScenarioName |
Enables tracing for a specific scenario |
Show Scenarios |
Shows a list of scenarios |
Conversations and Built-in Standard Filters
We've extended the idea of conversations to ETW logs. The conversation tree is populated with activities and providers under that which will help you narrow down interesting traffic. We also provide some standard filters to help you do some basic searches. And as usual, we try to propagate the important information to top of the tree.
And finally the default columns that show up in Network Monitor 3.3 are different when you open an ETW trace. We add a "UT Process Name" column, which stands for Unified Tracing, which displays possibly the process name, as well as the process ID. This replaces the "Process Name" column that shows up when you open capture files.
Where to Go Next?
When assisted support or further troubleshooting is necessary, the improved network trace logs can enable developers and support professionals to quickly isolate network activities and view the combined event data and packet captures in a single file, grouped by a network task and the related network activity.
On MSDN there's a great reference that discusses Win7 troubleshooting with Event Tracing (https://go.microsoft.com/fwlink/?LinkID=145404). This resource explains in greater detail how to troubleshoot problems, create filters, and use conversations with ETW tracing and Network Monitor. And while it's Windows 7 specific, there is still some applicability to older operating systems. And Network Monitor can read any ETL file. While we don't necessarily have parsers for every provider, that doesn't mean a parser can't be written.
Comments
Anonymous
January 01, 2003
If you are looking to follow this series, be sure to subscribe to my RSS feed at http://feeds.jasongaylord.com/JasonNGaylordAnonymous
January 01, 2003
If you are looking to follow this series, be sure to subscribe to my RSS feed at http://feeds.jasongaylordAnonymous
January 01, 2003
thanksAnonymous
January 02, 2014
Let’s start this new year with a small post on DirectAccess (DirectAccess remain in my good resolutionsAnonymous
May 16, 2015
Applies to: Windows Server 2012 R2 Windows 8.1 Windows Server 2012 Windows 8.0 Windows Server 2008 R2Anonymous
June 10, 2015
Applies to: Windows Server vNext Windows 10 Windows Server 2012 R2 Windows 8.1 Windows Server