Debugging a Debugger to Debug a Dump

Recently I came across an instance where my debugger did not do what I wanted.  Rarely do computers disobey me, but this one was unusually stubborn.  There was no other option; I had to bend the debugger to my will.

 

There are many ways to make a computer program do what you want.  If you have the source code you can rewrite and recompile the program.  If you have a hex editor you can edit the code of the binary.  A shim can be used to modify a program at runtime.  In this instance I was in a hurry and I was ok with a temporary solution, so I used a debugger to change the execution of the debugger while it ran.

 

Debug a debugger? Can you do such a thing?  Of course you can.

 

In this example a memory dump was captured of a system and I was asked to determine if the system had run out of desktop heap.  Usually the !dskheap command can be used to determine how much heap has been used by each desktop. Unfortunately, this command failed me.

 

23: kd> !dskheap

Error Reading TotalFreeSize from nt!_HEAP @ fffffa8019c65c00

Failed to GetHeapInfo for desktop @fffffa8019c65c00

EnumDsktps failed on Winsta: 19c4f090FillWinstaArray failed

 

The error indicates that the command couldn’t read from the _HEAP structure at fffffa8019c65c00 for desktop fffffa8019c65c00.  Further investigation found that reason I got this error is that the heap for the desktop in question is not valid memory.  Because the memory is described by a prototype PTE I assume that the heap has not been initialized (Note: See Windows Internals’ Memory Management chapter for more information about proto PTEs).

 

23: kd> dt win32k!tagDESKTOP fffffa8019c65c00

   +0x000 dwSessionId      : 0

   +0x008 pDeskInfo        : 0xfffff900`c05e0a70 tagDESKTOPINFO

   +0x010 pDispInfo        : 0xfffff900`c0581e50 tagDISPLAYINFO

   +0x018 rpdeskNext       : 0xfffffa80`19c6ef20 tagDESKTOP

   +0x020 rpwinstaParent   : 0xfffffa80`19c4f090 tagWINDOWSTATION

   +0x028 dwDTFlags        : 0x110

   +0x030 dwDesktopId      : 0x19c65c00`00000003

   +0x038 spmenuSys        : (null)

   +0x040 spmenuDialogSys  : (null)

   +0x048 spmenuHScroll    : (null)

   +0x050 spmenuVScroll    : (null)

   +0x058 spwndForeground  : (null)

   +0x060 spwndTray        : (null)

   +0x068 spwndMessage     : 0xfffff900`c05e0d90 tagWND

   +0x070 spwndTooltip     : 0xfffff900`c05e0fa0 tagWND

   +0x078 hsectionDesktop  : 0xfffff8a0`00ef08e0 Void

   +0x080 pheapDesktop     : 0xfffff900`c05e0000 tagWIN32HEAP

   +0x088 ulHeapSize       : 0x18000

   +0x090 cciConsole       : _CONSOLE_CARET_INFO

   +0x0a8 PtiList          : _LIST_ENTRY [ 0xfffffa80`19c65ca8 - 0xfffffa80`19c65ca8 ]

   +0x0b8 spwndTrack       : (null)

   +0x0c0 htEx             : 0n0

   +0x0c4 rcMouseHover     : tagRECT

   +0x0d4 dwMouseHoverTime : 0

   +0x0d8 pMagInputTransform : (null)

23: kd> dd 0xfffff900`c05e0000

fffff900`c05e0000  ???????? ???????? ???????? ????????

fffff900`c05e0010  ???????? ???????? ???????? ????????

fffff900`c05e0020  ???????? ???????? ???????? ????????

fffff900`c05e0030  ???????? ???????? ???????? ????????

fffff900`c05e0040  ???????? ???????? ???????? ????????

fffff900`c05e0050  ???????? ???????? ???????? ????????

fffff900`c05e0060  ???????? ???????? ???????? ????????

fffff900`c05e0070  ???????? ???????? ???????? ????????

23: kd> !pte fffff900`c05e0000

                                           VA fffff900c05e0000

PXE at FFFFF6FB7DBEDF90    PPE at FFFFF6FB7DBF2018    PDE at FFFFF6FB7E403010    PTE at FFFFF6FC80602F00

contains 000000076245C863  contains 0000000762569863  contains 000000045FA17863  contains F8A000F4F9780400

pfn 76245c    ---DA--KWEV  pfn 762569    ---DA--KWEV  pfn45fa17    ---DA--KWEV   not valid

                                                                                 Proto: FFFFF8A000F4F978

 

There are many desktops in this session and I wanted to know about the usage of the other desktops, but the !dskheap command stopped after just one error.  I needed to force it to continue after this error, so I launched a debugger to debug my debugger.  There is a command to do this, just run .dbgdbg.

 

23: kd> .dbgdbg

Debugger spawned, connect with

    "-remotenpipe:icfenable,pipe=cdb_pipe,server=NINJA007"

 

For clarity I will call the original debugger where I ran !dskheap debugger1, and the new debugger spawned by .dbgdbg debugger2 .

 

Before switching to debugger2 I need to know what I am going to debug.  The error message gives a hint about where to set a breakpoint, I am looking for a failure from GetHeapInfo.

 

23: kd> !dskheap

Error Reading TotalFreeSize from nt!_HEAP @ fffffa8019c65c00

Failed to GetHeapInfo for desktop @fffffa8019c65c00

EnumDsktps failed on Winsta: 19c4f090FillWinstaArray failed

 

I need to know which module GetHeapInfo is in, the .extmatch match command indicates which module contains the !dskheap command.

 

23: kd> .extmatch dskheap

!kdexts.dskheap

 

Switching to debugger2 I set a breakpoint on kdexts!GetHeapInfo.  Use Ctrl+C to trigger a debug break in cdb (this is the same as a Ctrl+Break in windbg).

 

0:004> bp kdexts!GetHeapInfo

0:004> g

 

Switch back to debugger1 and run the !dskheap command.

 

23: kd> !dskheap

 

In debugger2 I have hit the breakpoint.

 

Breakpoint 0 hit

kdexts!GetHeapInfo:

000007f9`4237b9b0 4055            push    rbp

 

The error says GetHeapInfo failed, so I am interested in what this function returns.  To see what GetHeapInfo returns I go up one level in the stack and set a new breakpoint on the code just after it returns.  This new breakpoint will also dump the return value of GetHeapInfo (return values are always in the rax register).

 

0:000> gu

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> r rax

rax=0000000000000000

0:000> bc *

0:000> bp 000007f9`4237b483 "r rax"

0:000> g

 

The next time the breakpoint hit the return value was 1, which in this instance means GetHeapInfo failed.  This is where I exerted my control over the computer: I forced the return value to 0.

 

rax=0000000000000001

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> r rax=0

 

I ran through the other breakpoints and changed rax as necessary.

 

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000001

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> r rax=0

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

 

Everything was going well, until the computer defied me again.  The !dskheap output computes the percentage of heap usage by dividing the bytes used by the size of the heap.  Unfortunately, the size of the heap was left at 0 for the two heaps where I changed the return value.  It is well known that only Chuck Norris can divide by zero; to prevent a roundhouse kick to your computer the processor generates an exception.

 

(2d0.928): Integer divide-by-zero - code c0000094 (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> r r11

r11=0000000000000000

0:000> r rax

rax=0000000000000000

0:000> g

 

Fortunately debugger1 handles the divide by zero exception and it is easy to run !dskheap again.

 

23: kd> !dskheap

 

Back in debugger2 I set a new breakpoint on the div instruction that outputs the divisor.  When the divisor (r11) is 0 I changed it to a non-zero value to avoid the wrath of Mr. Norris.

 

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> bp 000007f9`4237b90e

0:000> bp 000007f9`4237b90e "r r11"

breakpoint 1 redefined

0:000> g

rax=0000000000000001

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> r rax=0

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

rax=0000000000000001

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> r rax=0

0:000> g

rax=0000000000000000

kdexts!EnumDsktps+0x197:

000007f9`4237b483 4885c0          test    rax,rax

0:000> g

r11=0000000000033333

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> g

r11=0000000000000000

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> r r11=1

0:000> g

r11=00000000000007ae

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> g

r11=0000000000013333

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> g

r11=0000000000013333

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> g

r11=0000000000013333

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> g

r11=0000000000000000

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> r r11=1

0:000> g

r11=0000000000013333

kdexts!DisplayInfo+0x2ee:

000007f9`4237b90e 49f7f3          div     rax,r11

0:000> g

 

Finally, back in debugger1 I now have complete output for !dskheap.  After a few strategic modifications of the program’s execution I got it to output the data I wanted.  As it turns out we aren’t out of desktop heap after all.

 

23: kd> !dskheap

Error Reading TotalFreeSize from nt!_HEAP @ fffffa8019c65c00

Error Reading TotalFreeSize from nt!_HEAP @ fffffa801a53ea30

  Winstation\Desktop            Heap Size(KB)   Used Rate(%)

------------------------------------------------------------

  WinSta0\Default                  20480                 0%

  WinSta0\Disconnect                   0                 0%

  WinSta0\Winlogon                   192                 2%

  Service-0x0-3e7$\Default          7680                 1%

  Service-0x0-3e4$\Default          7680                 0%

  Service-0x0-3e5$\Default          7680                 0%

  Service-0x0-26f46a$\Default          0                 0%

  Service-0x0-2706f2$\Default       7680                 0%

------------------------------------------------------

                Total Desktop: (   51392 KB -   8 desktops)

                Session ID:  0

============================================================

Comments

  • Anonymous
    February 28, 2013
    This post is pure gold - useful and funny at the same time!