SOS: It's Not Just an ABBA Song Anymore

John Robbins

Code download available at:Bugslayer0306.exe(119 KB)


Using SOS
The Tips

Hidden deep inside the Microsoft® .NET Framework 1.1 is one of the more interesting features for extreme power users: Son of Strike (SOS). If your app is a pure managed code play, your development and debugging tasks are easily handled by existing Microsoft tools. However, if you're like most folks, you can't move over to the completely managed world instantly. You have a huge investment in COM or COM+ components that you need to leverage from your managed code. This border between managed and native code is where, as the old maps used to say, the dragons be.

Some of you might think that all you have to do is turn on mixed-mode debugging. Unfortunately, that doesn't help you see things like managed memory leaks, which can happen if you misuse the Framework Class Library (FCL). It also won't let you peer into a minidump file that you can create when the application crashes. Getting information from both the managed and native sides of your application used to be a supreme challenge consisting of long nights staring at enough x86 assembly language to drive you completely nuts.

Fortunately, Microsoft set out to solve the mixed programming issues with SOS. While they've done an admirable job, using SOS takes commitment because it's written as a WinDBG extension. Interestingly, starting with Visual Studio® .NET 2003, you can load SOS when doing mixed-mode debugging. However, I've found it easier to do from WinDBG. If you don't already have WinDBG, you can download it from Microsoft Debugging Tools. WinDBG is a hardcore debugger that handles everything from kernel mode to user-mode debugging all in a circa-1990 user interface. While the interface is a little hard to use, the power features of the debugger are outstanding. Since I don't have the space in this column to discuss WinDBG at length, I'll refer you to the WinDBG documentation or Chapter 8 of my book, Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press®, 2003). For the remainder of this column, I'll assume that you've read one of those aforementioned documents and therefore have become familiar with WinDBG.

The basic documentation for SOS is in the SOS.htm file in the \SDK\v1.1\Tool Developers Guide\Samples\SOS directory under your Visual Studio .NET installation path. If you open the documentation, you'll definitely see that "basic" is the operable term here. In essence, it consists of the list of commands in the SOS.DLL extension and a brief bit about their usage.

If you're dealing with larger systems based on the .NET Framework, especially heavy ASP.NET transactions, you'll also want to download the 170-page PDF file "Production Debugging for .NET Framework Applications,". If you want to know how to handle hung ASPNET_WP.EXE processes, deal with potential memory management issues in the .NET Framework, and take charge of other extreme-edge problems, this is an excellent document. The folks who wrote the document have definitely done their debugging of numerous live production systems, and their knowledge can save you quite a bit of hassle.

With these two documentation sources, you'll get a quick discussion of the SOS commands and an extreme hardcore tricks document, but there's still something missing: just how do you get started with SOS inside WinDBG? In this column, I want to help you get a leg up on SOS. My hope is that you'll get enough information out of this section to understand what's going on in the "Production Debugging for .NET Framework Applications" document. I won't be covering everything, such as all the garbage collector commands, because those are covered in detail in this longer document.

Before I start, I want to show you the easy way to get SOS.DLL loaded into WinDBG. SOS.DLL is part of the .NET Framework itself, so the trick involves getting the appropriate directories in your path so that WinDBG can easily load SOS.DLL. To do this you'll need to open an MS-DOS® command window and execute the file VSVARS32.BAT, which is located in the Visual Studio .NET\Common7\Tools directory. VSVARS32.BAT gets your environment set up so that all the appropriate directories for the .NET Framework are in your path.

Once you've executed VSVARS32.BAT, starting WinDBG from that MS-DOS command window allows you to load SOS.DLL simply by executing ".load sos" from the WinDBG Command window. WinDBG always puts the last loaded extension DLL onto the top of the chain, so executing "!help" from the command window shows you a quick listing of all the SOS.DLL commands.

Using SOS

Probably the best way to show SOS usage is with a live example. The ExceptApp program included with this column's sample files will show you how to get started with important commands (see the link at the top of this article). To keep things manageable, this code simply calls a few methods with local variables and finally throws an exception. I'll walk through an example of debugging EXCEPTAPP.EXE with SOS so that you can learn about the important commands for finding where you are when an application using managed code crashes or hangs. With that information, you'll be in a position to more easily apply SOS to the problems you'll encounter while being better able to understand the Production Debugging guide.

After you've compiled EXCEPTAPP.EXE and set up an MS-DOS command prompt as I described earlier, open up EXCEPTAPP.EXE in WinDBG and stop at the loader breakpoint. To make WinDBG stop when a common language runtime (CLR)-managed application throws an exception, you have to tell WinDBG about the exception number that the CLR throws. The easiest way to do this is to go into the Event Filters dialog box, click the Add button, and in the Exception Filter dialog box, enter 0xE0434F4D. Select "Enabled" in the Execution group box, and select "Not Handled" in the Continue group box. Once you click OK, you've successfully set WinDBG to stop whenever any EXCEPTAPP.EXE throws a .NET exception. If the value 0xE0434F4D looks somewhat familiar, you can always see what it stands for by using the .formats command.

After you have set the exception, run EXCEPTAPP.EXE until it stops on the .NET exception. WinDBG will report it as a first chance exception and stop the application on the actual Win32 API RaiseException call. After getting SOS loaded with a .load sos command, execute "!threads" (the first command you'll always want to execute in SOS) so that you can see which threads in the application or dump have CLR code in them. With EXCEPTAPP.EXE, the WinDBG thread command using the tilde character (~) indicates that three commands are running in the application. However, the all-important !threads command lists that only threads 0 and 2 have any .NET-compliant code in them, as shown in the output in Figure 1. (To get everything to fit in on the page, I show the individual thread information in a table. In WinDBG, you see it as a long horizontal display.)

Figure 1 !threads

0:000>!threads PDB symbol for mscorwks.dll not loaded succeeded Loaded Son of Strike data table version 5 from "e:\WINNT \Microsoft.NET \Framework \v1.1.4322 \mscorwks.dll" ThreadCount: 2 UnstartedThread: 0 BackgroundThread: 1 PendingThread: 0 DeadThread: 0 Row Heading WinDBG Thread ID 0 2 Win32 Thread ID 884 9dc ThreadObj 00147c60 001631c8 State 20 1220 PreEmptive GC Enabled Enabled GC Alloc Context 04a45f24:04a45ff4 00000000:00000000 Domain 00158300 00158300 Lock Count 0 0 APT Ukn Ukn Exception System.ArgumentException (Finalizer)

The important information you'll encounter in the !threads display consists of the Domain field, which tells you which AppDomains the threads are running in, and the Exceptions field, which happens to be overloaded. In the EXCEPTAPP.EXE example, the first thread has thrown the System.ArgumentException, so you can see the current exception for any thread. The third thread in EXCEPTAPP.EXE shows the special value (Finalizer), which indicates that the thread is, as you can guess, the finalizer thread for the process. You'll also see the values (Threadpool Worker), (Threadpool Completion Port), or (GC) in the Exception field. When you see one of those special values, you'll know they represent runtime threads, not your threads.

Since you've determined that the WinDBG thread 0 contains the EXCEPTAPP.EXE exception, you'll want to take a look at the call stack with !clrstack -all to see the full details of the stack, including parameters and locals (see Figure 2). Although !clrstack has switches to see the locals (-l) and parameters (-p), if you specify them together, they seem to cancel each other out and you see neither. If you'd like to walk all thread call stacks at once, you can use the command ~*e !clrstack.

Figure 2 !clrstack -all

Note, I excised the registers from this display 0:000> !clrstack -all Thread 0 ESP EIP 0012f5e0 77e73887 [FRAME: HelperMethodFrame] 0012f60c 06d3025f [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Doh (String,ValueClass ExceptApp.Days) at [+0x67] [+0x16] c:\junk\cruft\exceptapp\class1.cs:14 PARAM: this: 0x04a41b5c (ExceptApp.DoSomething) PARAM: value class ExceptApp.Days StrParam PARAM: unsigned int8 ValueParam: 0x07 0012f630 06d301e2 [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Reh (I4,String) at [+0x6a] [+0x2b] c:\junk\cruft\exceptapp\class1.cs:23 PARAM: this: 0x04a41b5c (ExceptApp.DoSomething) PARAM: class System.String i: 0x00000042 PARAM: int8 StrParam: 77863812 LOCAL: class System.String s: 0x04a45670 (System.String) LOCAL: value class ExceptApp.Days e: 0x003e5278 0x0012f63c •••

In the parameter display, there seems to be a bug because !clrstack doesn't always display the parameter types correctly. In the DoSomething.Doh method, you can see it takes a String and a Days value enumeration in the prototype. However, the PARAM: information shows the StrParam parameter as value class ExceptApp.Days and ValueParam as an unsigned int8. Fortunately for value parameters, even when the type is wrong, the correct value displays next to the parameter name. In the ValueParam example, the value passed in is 7, which corresponds to the enumeration "Fri."

Before I jump into figuring out the values of the value classes and objects, I want to mention another stack-walking command you might find useful. If you're dealing heavily with cross-.NET Framework and native calls and you'd like to see a call stack that includes everything, the !dumpstack command is your friend. Overall, it does a good job, but having full PDB symbols for the .NET Framework might make it better. Occasionally, the !dumpstack command reports "Use alternate method which may not work," which seems to indicate that it's attempting to walk the stack when it's missing certain symbol information.

In the LOCAL: display under the call to DoSomething.Reh are two local variables: s (a String object) and e (a Days value class). After each comes the hexadecimal address describing the type. For the Days value, there are two numbers: 0x003E5278 and 0x0012F63C. The first number is the method table and the second is the location in memory where the value is stored. Seeing the value in memory is simple using one of WinDBG's memory dumping commands such as dd 0x0012F63C.

Seeing the method table that describes the method data, the module information, and the interface map, among other things, is done through SOS's !dumpmt command. Executing !dumpmt 0x003E5278 with the EXCEPTAPP.EXE example gives the information shown in the following code:

0:000> !dumpmt 0x003e5278 EEClass : 06c03b1c Module : 001521a0 Name: ExceptApp.Days mdToken: 02000002 (D:\Dev\ExceptApp\bin\Debug\ExceptApp.exe) MethodTable Flags : 80000 Number of IFaces in IFaceMap : 3 Interface Map : 003e5380 Slots in VTable : 55

Looking at the method table, the first two numbers displayed show you which module a method comes from, as well as its execution engine class. For interfaces, the SOS documentation has an excellent example of how to walk the interface maps, and I would encourage you to look it over. If you have a burning desire to see all the methods in the vtable for a particular class or object along with their method descriptors, you can specify the -md option in front of the method table value. In the case of EXCEPTAPP.EXE's value class ExceptApp.Days, you'll see all 55 methods listed. As the SOS documentation mentions in the "How Do I... ?" section, getting the method descriptors is important to setting breakpoints on specific methods.

Since I'm looking at the class and module information for the ExceptApp.Days method table, I want to take a little detour. Once you have an execution engine class address, the !dumpclass command will tell you everything you ever wanted to know about a class, including all the data fields in the class. To see the information about a module, use the !dumpmodule command. The !dumpmodule output documentation has examples of how to walk through memory and find classes and method tables for a module.

Now that I've ground through the value class, let's make some sense out of the String local variable called "s", in DoSomething.Reh, which was displayed as follows:

LOCAL: class System.String s: 0x04a45670 (System.String)

Since s is an object, only one hexadecimal value is displayed after the variable name—the location of that object in memory. Using the !dumpobj command, you'll see all the information about that object (see Figure 3).

Figure 3 !dumpobj

0:000> !dumpobj 0x04a45670 Name: System.String MethodTable 0x79b7daf0 EEClass 0x79b7de3c Size 92(0x5c) bytes mdToken: 0200000f (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) String: Tommy can you see me? Can you see me? FieldDesc*: 79b7dea0 MT Field Offset Type Attr Value Name 79b7daf0 4000013 4 System.Int32 instance 38 _arrayLength 79b7daf0 4000014 8 System.Int32 instance 37 m_stringLength 79b7daf0 4000015 c System.Char instance 54 m_firstChar 79b7daf0 4000016 0 CLASS shared static Empty >> Domain:Value 00158298:04a412f8 << 79b7daf0 4000017 4 CLASS shared static WhitespaceChars >> Domain:Value 00158298:04a4130c <<

As you can see from the output, some of the fields—MethodTable, EEClass, and MT (Method Table)—can be used with commands I've previously discussed. For the field members, the !dumpobj command will show the values directly in the table for simple value types. In the String display in the output in Figure 3, the m_stringLength value is the 37 characters currently in the string. As you'll see in a moment, for object field members, the Value field will contain the object instance, and you can use the !dumpobj command to see the value.

The entries delineated by >> and << are showing you the domain instance and location in that domain for the static field prior to the >>. If I had multiple AppDomains in EXCEPTAPP.EXE, you'd see two domains and value information output for the static WhitespaceChars field.

Now that I've covered some of the basic commands, let's tie them together and see how you can look up useful data with them. With EXCEPTAPP.EXE stopped in WinDBG because of an exception, it would be nice to see what the exception is and what some of the fields are so you can see why EXCEPTAPP.EXE stopped in the middle of execution.

You know from executing the !threads command that the first thread is currently processing an exception, System.ArgumentException. If you look carefully at the output for !clrstack or !dumpstack, you'll notice that no locals or parameters that show any type of System.ArgumentException are displayed. The good news is that an excellent command, !dumpstackobjects, shows all the objects that are currently on the stack of the current thread, as you can see in the output in Figure 4.

Figure 4 !dumpstackobjects

0:000> !dumpstackobjects ESP/REG Object Name ebx 04a45670 System.String Tommy can you see me? Can you see me? 0012f50c 04a45f64 System.ArgumentException 0012f524 04a45f64 System.ArgumentException 0012f538 04a45f64 System.ArgumentException 0012f558 04a44bc4 System.String Reh = 0012f55c 04a45f64 System.ArgumentException 0012f560 04a45670 System.String Tommy can you see me? Can you see me? 0012f564 04a4431c System.Byte[] 0012f568 04a43a58 System.IO.__ConsoleStream 0012f5a0 04a45f64 System.ArgumentException •••

Since the !dumpstackobjects command is wandering up the stack, you'll see some items multiple times as they are passed a parameter to multiple functions. In Figure 4, you can see multiple System.ArgumentException objects, but if you look at the object value next to each object, you'll notice they are all referring to the same object instance, 0x04A45F64.

To inspect the System.ArgumentException object, I'll use the !dumpobj command. Note that I had to wrap the Name column to get everything to fit on the page (see Figure 5).

Figure 5 System.ArgumentException Object

0:000> !dumpobj 04a45f64 Name: System.ArgumentException MethodTable 0x79b87b84 EEClass 0x79b87c0c Size 68(0x44) bytes mdToken: 02000038 (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) FieldDesc*: 79b87c70 MT Field Offset Type Attr Value Name 79b7fcd4 400001d 4 CLASS instance 00000000 _className 79b7fcd4 400001e 8 CLASS instance 00000000 exceptionMethod 79b7fcd4 400001f c CLASS instance 00000000 _exceptionMethodString 79b7fcd4 4000020 10 CLASS instance 04a456cc _message 79b7fcd4 4000021 14 CLASS instance 00000000 _innerException 79b7fcd4 4000022 18 CLASS instance 00000000 _helpURL 79b7fcd4 4000023 1c CLASS instance 00000000 _stackTrace 79b7fcd4 4000024 20 CLASS instance 00000000 _stackTraceString 79b7fcd4 4000025 24 CLASS instance 00000000 _remoteStackTraceString 79b7fcd4 4000026 2c System.Int32 instance 0 _remoteStackIndex 79b7fcd4 4000027 30 System.Int32 instance -2147024809 _HResult 79b7fcd4 4000028 28 CLASS instance 00000000 _source 79b7fcd4 4000029 34 System.Int32 instance 0 _xptrs 79b7fcd4 400002a 38 System.Int32 instance -532459699 _xcode 79b87b84 40000d7 3c CLASS instance 04a45708 m_paramName

Inside an exception, the Message property is the important property. Since I can't call a method directly from WinDBG to see its value, I'll have to look at the _message field, which is where the Message property stores the actual string. Since this field is marked with CLASS, the hexadecimal number in the Value column is the object instance. To look at the object, I'll do another !dumpobj command to view it. As you've seen, the String object will have a special field, so you can see its actual value, which turns out to be the innocuous "Throwing an exception".


Given the complexity of SOS, I hope I was able to bridge the gap between the basic documentation and the "Production Debugging for .NET Framework Applications" document. SOS is one of those tools that you hope you'll never have to use, but if you do have to use it sometime, you want to make sure that you have played with it before you're faced with that showstopper bug. Trying to learn SOS and debug at the same time is a recipe for sleeping in your office for two straight weeks.

The Tips

Spring is the time of love and I'd love to have you send your debugging tips to john@wintellect.com.

Tip 55 To attach to a process in Visual Studio .NET for quick native debugging, hold down the control key when you click Attach in the Processes dialog. This will bypass the Attach To Process dialog and immediately attach for native debugging only.

Send your questions and comments for John to  slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in programming for .NET and Windows. This column was adapted from a chapter in Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at https://www.wintellect.com.