Enjoy!

Enjoy!

If you are involved in interoperating with Microsoft products, these Test Suites can provide some valuable information. Also be sure to check out all of our Interoperability Testing Links.

You’ve seen a lot of updates lately on Experts with the move to the Outercurve Foundation, but we have a new expert for you too! It’s the brethren of Top Users: Top Protocols. Download it here over on CodePlex.

What is Top Protocols?

Top Protocols is another simple expert designed to give you a high-level summary of what’s occurring in a trace. While Top Users shows you the chattiest boxes on a network, likewise Top Protocols shows you the chattiest Protocols.

Once installed, it works like any other Expert in the Experts menu. Just find NMTopProtocols in the list and select Run Expert. The data will automatically start being parsed as indicated in the status bar:

What can it do?

Top Protocols walks through your trace and constructs a count of the highest-level protocols it encounters. It counts the number of times it sees each one, the structure for how that protocol was found in the stack and the number of bytes each frame of information contained. It can also reassemble the data as it goes so fragments get counted as the initiating protocol instead of just raw transport data.

One important note here is that Top Protocols only parses as far as it can using the selected Parsing Profile from Network Monitor. So, if you want the full breadth of parsers, you’ll want to make sure to be on the Windows profile first. If you just want a quick summary, change your profile first, and Top Protocols will load a little quicker.

After it’s collected this information, it displays it in three main views in the tabbed interface:

Note: you’ll need the MS Chart Controls pointed to in the download page in order to use the Pie Chart and Time Graph modes.

Overview

The overview is a basic table of the raw data collected. By default it’s sorted by the breakdown of the Protocol tree (which you can also see on the left), but you can use the columns at the top to sort by the different data points such as number of frames or number of bytes.

If after you’ve done some sorting you want to go back to the initial view, you can find the Restore Default Ordering option in the context menu of the grid:

If you select a node in the tree on the left, your view will be filtered to see only that protocol and its children:

Pie Chart View

The Pie Chart view is a recent addendum to the tool and is pretty simple at the moment. It gives you a quick and dirty way to visually see the most prevalent protocol in the trace:

Here we can see that HTTP traffic is the primary component to this trace.

This view also filters out the noise and only displays those elements which are end nodes of the tree.

Time Graph View

This view is the most exciting of all in Top Protocols. It’s almost like another expert in itself, and you get it for free!

The Time Graph view shows how the protocol traffic was received over the course of time the trace was taken. This lets you see spikes in protocol traffic to more easily determine when data was sent or received.

By default this mode filters out the protocols that don’t meet a certain threshold. This threshold can be configured in the options menu, as well as the scale of the graph.

Helpful Tricks

There are a lot of settings in Top Protocols, which can be found under the options menu.

The first option simply tells Top Protocols whether or not to reassemble the data first, thus any fragments encountered later are counted towards the tally of the initial protocol. Therefore, if you had a HTTP payload spanning multiple packets, each one would tally towards the total of HTTP traffic encountered. With this option off, you’d see only HTTP counts for the headers and the rest would be lumped under TCP. This could mask the intensity of certain types of traffic when turned off, but the comparison can tell you how fragmented your data is and if it was more requests with smaller payloads or less requests with larger payloads. A work item for the future could be to calculate both of these values at the same time and present that data as well.

The next set of options refers to the Time Graph and how it is calculated. The first option changes the number of intervals used to segment the data. The more slices used the greater the resolution to see changes but the more memory which will be required. You can also determine how much data needs to occur for a protocol to automatically be selected in the graph.

Use the “Show Tree as Hierarchy” option to decide if the data is grouped under their carrier protocols or not. And use the following three options to determine whether or not to use certain filters available from Network Monitor.

And lastly, you can decide how all these settings are persisted.

One thing to note with all these settings though, is that they’ll only take effect the next time a file is loaded. However, you can quickly reload your current view using the ‘Reload File’ option in the File menu:

Why’s this version 3.2 and what’s next?

Top Protocols has been around for a while, but started out like most of our experts as an internal project. Since then, we’ve started pushing Experts into the community and now as part of the Outercurve foundation. This makes it a lot easier to work on these projects for everyone’s benefit as Paul’s explained before.

Originally Top Protocols was written by Paul and ran on the command prompt. Michael took the project when the Experts feature was introduced and re-wrote the project to include more features and the GUI. It’s been through a couple of revisions since, but that’s why it’s already up to v3.2 when it was transferred to Outercurve and made available to everyone.

As for what’s next, I recommend you check out the Top Protocols homepage. We welcome people to submit issues and ideas and if you’re adventurous enough to hop on board in helping out. And we’ll see where we go from there together.

What’s New with Top Users

The biggest update is that there is now only one version. You can switch between the Endpoint and Conversation mode from within the expert. You also have the ability to open new captures directly from the tool. Here’s a list of the other changes:

• Conversation mode will show a percentage when viewing a single address type, like Ethernet or IPv4.
• The sorting has been improved when changing address types.
• There was a bug in the send/receive statistics that is now fixed.
• Command line arguments are case insensitive now.
• The DumpRow command line feature wasn’t working and is now fixed.
• Data is now right justified properly.
• Formatting for numeric data has been improved.
• Address columns are auto sized correctly.
• Status bar shows start/end frame times for entire trace.
• Converted to VS2010

Bob’s Footsteps

To get Bob involved, we first had him sign a typical open source document electronically. This basically is to protect the Outercurve Foundation. After this we discussed on the phone the changes he was proposing because he wanted to make sure they were sound. We added Bob as a contributor to the CodePlex project, and the first step he made was to branch the project. Once his changes were made in the branch, I synced and verified that the changes made sense with regards to the original architecture. I also made some small modifications to the version and build process. Then we merged the project back and we were done.

Contributions to any of our projects can be as little as a feature request or bug report.  If you want to go further you can suggest code to fix or address a feature.  When you get more involved in a project there are opportunities like becoming a contributor or even coordinator.  Issue reports and discussions can be started for the project you are interested in. The list of projects is listed on the main CodePlex Project page. Also, if you have your own ideas for Experts you'd like to get started with in open source; we'd welcome you to join us as well. Just drop us a line by going to our Outercurve Project Page, and send email to the project lead.

With a Little Help from my Friends

It’s awesome to see contributions, like these from Bob Sledge, for a tool that is used by many. This is exactly the reason we wanted to make these experts open source. Perhaps this will inspire others to put their two cents in.

Now You Can Select a IPv4/IPv6 Conversation

Previously you had to select the specific TCP conversation in order for the expert to work. This TCP conversation also had to contain the full TLS/SSL session setup, because the expert needs this data to do the decryption. However there are cases where a single IE session would spawn multiple TLS/SSL sessions, with only the first session containing the TLS/SSL session setup. Now, the expert can work in situations where multiple SSL sessions exist with two given restrictions:

1. The trace can only contain traffic that is associated with the private key you’ve provided. If there is other traffic that happens simultaneously, the expert will fail. Community: It should be possible to compare the certificate path provided with the one used during negotiation. With this mapping you could potentially decrypt all traffic that uses the same certificate. Perhaps we could also just ignore the failures and continue on.
2. Sessions can’t be intermingled. When the expert decrypts traffic, there’s a running check called a MAC (Message Authentication Code). State between separate TCP conversations is not maintained. So data from another TCP session messes up the MAC calculation and the expert fails. The work around for this problem is to manually save each TCP session in order to another file. This is simple using the Frame Buffer manager feature in the UI. Just select each TCP conversation, Ctrl+A to select all frames, and right click to open the Frame Buffer manager. Once you create a new file continue to add frames from each session in order to the same file, and then close the file once you are done. Community: While it might be hard to maintain state in the expert, I think it would be simple to make multiple passes though the capture file; limiting each pass to a single TCP conversation.

Support for Alternate Network Paths

The expert examines the traffic and derives a fully qualified path from the frame it’s examining. For instance if the path it sees is Ethernet.IPv4.IPv6.TCP, it uses this as a seed to understand the root for the TLS/SSL specific data. However, each of these is verified against known paths. In order to support new paths, I’ve added a few more common paths for wireless and ESP over IPv4/IPv6. Incidentally there was also a bug fix for IPv6 traffic that I fixed in this area. Community: If there are other network paths that we need to support, this is a very easy change. Also related to this is the support for TLS/SSL for other protocols. Currently we support LDAP and HTTP, but this would be easily extended to support other protocols.

TCP Fragmented Data

The Expert depended on the ability of Network Monitor parser to properly re-assemble TCP data. However, there are certain limitations where data does not align on a TCP frame boundary. This is because TCP streams data and therefore it’s possible for the data to break on non-frame bounds. However, from the TLS/SSL perspective, the data that describes each data segment is described by the protocol. So I modified the expert to find when there was not enough data to account for a full TLS/SSL Segment. In those cases, I reassembled that data manually. Community: There are bound to be problems between various versions of TLS/SSL. My main test case was TLS 1.0, so be aware for variances in other protocols that could cause issues.

Get Involved

The link to the latest version of the expert is available on our Expert Codeplex site. By highlighting some of the remaining issues I’m hoping to get more community involvement to further update this expert. And with the move to the OuterCurve Foundation contributing has never been easier.

1. NMDecrypt – Decrypts SSL data, given the private key.
2. NMTopUsers – Displays the top talkers on your network.
3. NMTopProtocols – Displays protocol distribution.

Additionally we’ve moved the Network Monitor SDK which contains helper code for experts you create.

Why the Outercurve foundation?

The Outercurve Foundation, previously known as the CodePlex Foundation, defines and maintains a contribution framework and simple licensing terms. These projects will use the BSD license, a familiar open source license that many of you are comfortable with. The foundation allows you to contribute to these projects easily, after executing a simple DocuSign contract.

Get Involved

If you have some good ideas for improvement to our Network Monitor experts or ideas for new Experts, we’d love to have you join our project as a contributor. We’ve already heard from Bob Sledge of Beta-Signma Data Systems who has expressed interesting in making some useful updates to the NMTopUsers expert. He is now signed up through the Outercurve Foundation and is ready to help us out. To get involved you can contact us via the Outercurve page for our Network Monitor Experts Project. By contributing to these open source projects, we can improve and create tools to makes all of our lives easier and engage the power of the community and Open Source.

My Setup

In my case I simplified and tested this from within my private wireless network. The Smart Potato app has a settings section which allows you to change, among other things, the server URL. So I modified this field to contain the private address for my Media Center, https://192.168.1.7. When I attempted to connect from Smart Potato, I received an error message that I could not find the server.

Getting a Trace

I started a network trace on my Media Center PC. I assumed all traffic should be directed there, but I suppose it’s possible this was not the case. However it is easy to start by sniffing at the Media Center PC and if I didn’t see traffic I could move up to the router. I also looked at my router to see what my Windows Phone 7 IP address was. I could also have looked in the Setting on my phone for my wireless connection as it shows up there as well.

Using this IP address I applied the following Display Filter so that I would only see traffic from or to my phone.

IPv4.Address == 192.168.1.8

I could have instead used a capture filter because there might have been other traffic that emanated from my Media Center to another service that might be relevant. If I instead used a capture filter, I would have to rerun my test to capture this data. But with no capture filter I still capture everything, and narrow down my view by applying various display filters after the fact.

Starting My Test

So the first thing I noticed when I started my capture is that I saw a TCP Syn request on port 80 from my phone. But the traffic looks like this:

Source	Destination	Description
192.168.1.8	192.168.1.7	TCP:Flags=......S., SrcPort=50914, DstPort=HTTP(80), PayloadLen=0, Seq=1438303684, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192
192.168.1.8	192.168.1.7	TCP:[SynReTransmit #2]Flags=......S., SrcPort=50914, DstPort=HTTP(80), PayloadLen=0, Seq=1438303684, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192
192.168.1.8	192.168.1.7	TCP:[SynReTransmit #2]Flags=......S., SrcPort=50914, DstPort=HTTP(80), PayloadLen=0, Seq=1438303684, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192

What we see is that the Media Center PC never responded to the request. Each subsequent request is flagged with the SynReTransmit keyword. If I wanted to look for these specifically, I could use a filter of:

Property.TCPSynReTransmit

For this traffic, I concluded that either the service was not listening on port 80 or perhaps it was getting blocked by my firewall. I enabled and then checked my firewall logs and saw that nothing was logged, so my firewall was letting that traffic through.

Next I used NetStat –ano, which list out all ports in use and their associated process Ids.

C:\Windows\System32\drivers\etc>netstat -ano
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 848
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:554 0.0.0.0:0 LISTENING 4200
TCP 0.0.0.0:954 0.0.0.0:0 LISTENING 5944
TCP 0.0.0.0:2869 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING 1292
TCP 0.0.0.0:3390 0.0.0.0:0 LISTENING 1292
TCP 0.0.0.0:5357 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:9080 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:10243 0.0.0.0:0 LISTENING 4
…

Fixed Port, but Now What?

I then modified the URL to say “https://192.168.1.7:9080”. This will direct the URL to use an alternate port instead of the default of 80. And when I checked Smart Potato after a reinstall, I found that the default string which I hastily replaced also ended with 9080. I suppose I could have paid more attention when changing the original URI.

After updating the port I tried again, but this time it reported bad username or password. I decided that since I had everything setup, I would trace again. This is the traffic I saw.

192.168.1.8	192.168.1.7	TCP:Flags=......S., SrcPort=53492, DstPort=9080, PayloadLen=0, Seq=793206766, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192
192.168.1.7	192.168.1.8	TCP:Flags=...A..S., SrcPort=9080, DstPort=53492, PayloadLen=0, Seq=4192041810, Ack=793206767, Win=8192 ( Negotiated scale factor 0x2 ) = 32768
192.168.1.8	192.168.1.7	TCP:Flags=...A...., SrcPort=53492, DstPort=9080, PayloadLen=0, Seq=793206767, Ack=4192041811, Win=64 (scale factor 0x8) = 16384
192.168.1.8	192.168.1.7	HTTP:Request, GET /xml/login, Query:un=paul&pw=password
192.168.1.7	192.168.1.8	HTTP:Response, HTTP/1.1, Status: Ok, URL: /xml/login

You will notice that in the Request frame, there is a username and password in plain text. (BTW, I changed the password to “password”). The fact that it’s an HTTP request tells me that this user name and password were probably not the Machine’s credentials as I assumed when I entered them onto application settings. Once again I looked at the Remote Potato settings and noticed it also had a security settings box. I reset this information on my phone and tried again and now I was able to connect.

Have My Potato and Watch It

Now I’m able to stream my Media Center recorded shows to my phone, which is pretty cool. If I’m on the road, in a plane, or somewhere remote I can schedule something to record, and then using Smart Potato, watch it immediately. Using Network Monitor to view data, and focusing on the IP address, will often give you some interesting information and is a useful way to spy on a device.