Matt's Blog https://docs.microsoft.com/archive/blogs/meason/feed.xml OS development for the HoloLens Tue, 16 Apr 2019 05:51:08 GMT en-US hourly 1 Setting debugger target architecture: .effmach https://docs.microsoft.com/archive/blogs/meason/setting-debugger-target-architecture-effmach Fri, 12 May 2017 09:53:21 GMT https://blogs.msdn.microsoft.com/meason/?p=315 0:000> k # Child-SP RetAddr Call Site 00 00000000`0063e958 00000000`6c8d210d wow64cpu!<redacted>+0xc 01 00000000`0063e960 00000000`6c88bfa1 wow64cpu!<redacted>+0xc 02 (Inline Function) --------`-------- wow64!<redacted>+0xd 03 00000000`0063ea10 00000000`6c87cbb0 wow64!<redacted>+0xf311 04 00000000`0063ea90 00007ffd`d1ef2a11 wow64!<redacted>+0x120 05 00000000`0063ed40 00007ffd`d1f28986 ntdll!LdrpInitializeProcess+0x1551 06 00000000`0063f140 00007ffd`d1ed9fae ntdll!_LdrpInitialize+0x4e982 07 00000000`0063f1c0 00000000`00000000 ntdll!LdrInitializeThunk+0xe When this is encountered, it means your debugger is interpreting the data streams with the wrong processor architecture.  Simply set it to the target architecture (x86), and you'll start to see the expected callstacks: 
0:000> .effmach x86
Effective machine: x86 compatible (x86)
0:000:x86> kpnL
 # ChildEBP RetAddr 
00 0067f8cc 763c2e1a ntdll_77130000!NtDelayExecution(void)+0xc
01 0067f934 763c2d7f KERNELBASE!SleepEx(unsigned long dwMilliseconds = 0x1, int bAlertable = 0n0)+0x8a
02 0067f944 006c58b0 KERNELBASE!Sleep(unsigned long dwMilliseconds = 0x1)+0xf
03 0067fa44 006b6762 foo_6a0000!<redacted>
04 0067fb44 006b6f22 foo_6a0000!<redacted>
05 0067fbc0 73f462c4 foo_6a0000!<redacted>
06 0067fbd4 77190fd9 kernel32!BaseThreadInitThunk(<redacted>)+0x24
07 0067fc1c 77190fa4 ntdll_77130000!__RtlUserThreadStart(<redacted>)+0x2f
08 0067fc2c 00000000 ntdll_77130000!_RtlUserThreadStart(<redacted>)+0x1b
]]> Kd breakpoints don't persist through reboot https://docs.microsoft.com/archive/blogs/meason/kd-breakpoints-dont-persist-through-reboot Tue, 28 Feb 2017 17:25:59 GMT https://blogs.msdn.microsoft.com/meason/?p=305 Debugging LoadLibrary Failures https://docs.microsoft.com/archive/blogs/meason/debugging-loadlibrary-failures Wed, 09 Nov 2016 18:00:52 GMT https://blogs.msdn.microsoft.com/meason/?p=295 covered pretty well, but it is worth repeating: If you are seeing a ERROR_MOD_NOT_FOUND (0n126, 0x7E, 0x8007007E) failure during a LoadLibrary, make sure the DLL in question is in the DLL search path.  If it is in the DLL search path, then the next likely step is to determine if one of that DLL's load-time dependencies are missing.  To diagnose this: Option 1: Loader Snaps Log (enabled via GFlags.exe): Enable loader snaps via gflags and re-run your scenario under a debugger.  You'll get swamped in debug spew, but a quick search for "ERROR" will yield the culprit.  The most likely failure spew will consist of 0xc0000135 (STATUS_DLL_NOT_FOUND). Option 2: Fusion Log Viewer: You can use the loader snaps method above (gflags), but there's another option here that doesn't require a debugger: Fuslogvw.exe.  It ships with Visual Studio and can be quickly accessed via the VS console.  Make sure to run this as admin to get everything logged. For more information on Fuslogvw.exe, see this URL.  ]]> SOS Versioning with Windbg https://docs.microsoft.com/archive/blogs/meason/sos-versioning-with-windbg Wed, 05 Oct 2016 14:15:47 GMT https://blogs.msdn.microsoft.com/meason/?p=275 0:000> .load C:\Debuggers\Psscor4\x86\x86\psscor4.dll ---------------------------------------------------------------------------- The user dump currently examined is a minidump. Consequently, only a subset of sos.dll functionality will be available. If needed, attaching to the live process or debugging a full dump will allow access to sos.dll's full feature set. To create a full user dump use the command: .dump /ma <filename> ---------------------------------------------------------------------------- 0:000> !dumpheap Failed to find runtime DLL (clr.dll), 0x80004005 Extension commands need clr.dll in order to have something to do. 0:000> .chain Extension DLL search Path:     <snipped for brevity> Extension DLL chain:     C:\Debuggers\Psscor4\x86\x86\psscor4.dll: image 4.0.0.4, API 1.0.0, built Tue Feb  1 12:39:38 2011         [path: C:\Debuggers\Psscor4\x86\x86\psscor4.dll]     <snipped for brevity> 0:000> .unload C:\Debuggers\Psscor4\x86\x86\psscor4.dll Unloading C:\Debuggers\Psscor4\x86\x86\psscor4.dll extension DLL The fact that the DLL was built in 2011 should have been a hint that it's waaaay outdated, but the not-so-super-helpful error message of 0x80004005 led me astray for a while. A common tactic is for folks to let the system figure out where the correct SOS.dll is, by using the .loadby command.  However, that doesn't help on the HoloLens dumps I look at, since my dev machine is not running a HoloLens.  Again, the debugger spits out a not-so-super-helpful error message as before: 
0:000> .loadby sos clr
Unable to find module 'clr'
The keys to get this to work are:
  1. Explicitly load the correct sos.dll version
  2. Make sure your symbol path is set appropriately
You can find the version by examining file properties, on the details tab.  Look at the version your dumped process is running, and find the appropriate sos.dll to go with it.  Here's an example: 
0:000> .load c:\foo\sos.dll
----------------------------------------------------------------------------
The user dump currently examined is a minidump. Consequently, only a subset
of sos.dll functionality will be available. If needed, attaching to the live
process or debugging a full dump will allow access to sos.dll's full feature
set.
To create a full user dump use the command: .dump /ma <filename>
----------------------------------------------------------------------------
0:000> !dumpheap

// Lots of debugger spew snipped for locating the CLR symbols, snipped here due to internal Microsoft stuff
Your best bet is to always use the latest SOS.dll that ships with the WDK.    ]]> Driver debug breakpoint https://docs.microsoft.com/archive/blogs/meason/driver-debug-breakpoint Wed, 21 Sep 2016 16:04:23 GMT https://blogs.msdn.microsoft.com/meason/?p=265 DebugBreak() API is the primary way to implement a breakpoint through code.  It's great to use when developing a prototype and exploring your environment. User mode developers (such as myself) may not be aware that this can also be used from a kernel mode driver.  This enables us to explore some aspects of kernel mode behavior as it relates to our user mode code. I use a custom-built driver whose sole purpose is to respond to IRP_MJ_DEVICE_CONTROL.  It has the basic driver packaging for registering and unregistering itself, but other than that, its sole purpose is to respond to IOCTL requests and do my bidding. I use a Win32 app to install it through CreateService() with SERVICE_KERNEL_DRIVER.  After it is installed, I use CreateFile() to get a handle to the driver object and DeviceIoControl() to invoke it. Once your breakpoint is hit, be sure to set process context to your user mode process, in order to see symbols for the user mode portions of your call stack: 
3: kd> !process -1 0
PROCESS afbe7040  SessionId: 0  Cid: 0cfc    Peb: 03213000  ParentCid: 0cf8
    DirBase: 7ffe0660  ObjectTable: b22e4480  HandleCount:  38.
    Image: MeasonTest.exe
3: kd> .process /p /r afbe7040
Implicit process is now afbe7040
.cache forcedecodeuser done
Loading User Symbols
DBGENG: PDB symbols loaded for 'ntdll.dll'
3: kd> k
 # ChildEBP RetAddr  
00 a8fa0bc0 81cadf43 MeasonTestDrv!DispatchDeviceControl+0x9f
01 (Inline) -------- nt!(redacted)
02 a8fa0bdc 81ef150c nt!(redacted)
03 (Inline) -------- nt!(redacted)
04 (Inline) -------- nt!(redacted)
05 a8fa0c30 81ef0f80 nt!(redacted)
06 a8fa0cf8 81ef0b8a nt!(redacted)
07 a8fa0d24 81da4097 nt!(redacted)
08 a8fa0d24 77ec2090 nt!(redacted)
09 00edf940 77ec0a8a ntdll!KiFastSystemCallRet
DBGENG: PDB symbols loaded for 'KERNELBASE.dll'
0a 00edf944 77aee0fb ntdll!NtDeviceIoControlFile+0xa
DBGENG: PDB symbols loaded for 'MeasonTest.exe'
0b 00edf9a4 011db2f0 KERNELBASE!DeviceIoControl+0x7b
0c 00edfa20 011db1b6 MeasonTest!DriverInvoke::InvokeDriver+0x40
0d 00edfa30 011dfdf8 MeasonTest!DriverInvoke::DebuggerBreak+0x16
0e 00edfb04 011e0fbf MeasonTest!main+0x128
0f 00edfb48 77e51154 MeasonTest!__mainCRTStartup+0x107
10 00edfb90 77e51114 ntdll!__RtlUserThreadStart+0x3a
For those just now learning driver development, this is also a great way to test out DDK API calls or explore kernel space.]]> Sample debugging session without symbols https://docs.microsoft.com/archive/blogs/meason/sample-debugging-session-without-symbols Fri, 26 Aug 2016 10:09:26 GMT https://blogs.msdn.microsoft.com/meason/?p=255    hr = foo->bar(value1, static_cast<unsigned long>(value2)))) When I first saw the code, I assumed the foo variable was nullptr. However, it was a valid pointer. Where was the nullptr dereference being made? The faulting call stack terminated with the following frames: 
0072dda0 100260f5 00b8737c 5e9365d8 00000091 0x0
0072dddc 10027f57 51be7d60 5197d7b8 5197d788 foo!baz+0x30e5
0072ddfc 100267de 00000000 00000000 00000000 foo!baz +0x4f47
0072de2c 10025c2e 0072de7c 0072de58 1d6507fc foo!baz +0x37ce
0072de98 10025951 00b8737c 0072dec0 0072e7e8 foo!baz +0x2c1e
0072e6c4 555fb5fb 008189f8 00b8737c 0072e914 foo!baz +0x2941
Using the windbg ‘ub’ command, I dumped out the x86 ASM that was executing up to the point of the exception: 
mov    eax,dword ptr [ebp-1Ch]
push   dword ptr [ebp-0Ch]
push   esi mov    eax,dword ptr [eax+4]
push   eax mov    ecx,dword ptr [eax]
mov    esi,dword ptr [ecx+30h]
mov    ecx,esi
call   esi // This is calling into the ‘bar’ method
Using the ChildEBP for the frame, the annotated instruction flow looks like: 
// ebp = 0072dddc (from faulting call stack)
mov     eax,dword ptr [ebp-1Ch] // eax = *(0072dddc - 0x1c) = *(0072ddc0) = 0072de7c = this
push    dword ptr [ebp-0Ch]     // push *(0072dddc - 0xc) = *(0072ddd0) = 00000091 = value2
push    esi                     // push esi = 0x5e9365d8 = value1
mov     eax,dword ptr [eax+4]   // eax = *(eax+4) = *(0072de7c+4) = *(0072de80) = 00b8737c = this->foo
                                // where: *this->foo = blah_555e0000!xyz+0x135fe
push    eax                     // push eax = 00b8737c
mov     ecx,dword ptr [eax]     // ecx = *(eax) = *(00b8737c) = 55604874 = blah_555e0000!xyz+0x135fe
mov     esi,dword ptr [ecx+30h] // esi = *(55604874+0x30) = *(556048a4) = 00000000
mov     ecx,esi
call    esi
The stack looked like it was set up correctly: 
0:000> dd 0072dda0
0072dda0 00000091 100260f5 00b8737c 5e9365d8     // blah_555e0000!xyz+0x135fe=5e9365d8
0072ddb0 00000091 5197d7b8 5197d788 00000000     // value2=0x91, value1=0x5197d7b8
0072ddc0 0072de7c 5e9365d8 51be7df8 00000000
0072ddd0 00000091 00000000 1d650460 0072ddfc
0072dde0 10027f57 51be7d60 5197d7b8 5197d788
0072ddf0 057684b8 00000000 00b873b0 0072de2c
0072de00 100267de 00000000 00000000 00000000
0072de10 00b83538 5197d788 0072de60 05750048
This code is dereferencing a vtbl offset. Thus, I suspected the contracts between the two sides of software were not in agreement. When I ran my local version of this code, where I know the 2 sides of software agree on the software contract, that same line of code that crashes instead set esi to be the entry instruction for blah!bar (where the class implementing the interface comes from the ‘blah’ module). Using the windbg ‘dds’ command, I was able to dump out the symbolic vtbl for the class. This confirmed that the method in question was indeed at offset 0x30 from the base address: 
0:000> dds 00224e28
// … extraneous data omitted
00224e58 00240310 blah!bar
My suspicion turned out to be correct. The team that had the symbols was able to look up the vtbl, and confirmed that our software contracts were not in agreement. Besides debugging skills, what we can learn from this in software principles and design?  When your software is interacting with external code, make sure you either have a contract that doesn't change, or has a solid protocol for negotiating the differences.]]> Intro to kernel debugging 3 https://docs.microsoft.com/archive/blogs/meason/intro-to-kernel-debugging-3 Thu, 07 Jul 2016 17:27:11 GMT https://blogs.msdn.microsoft.com/meason/?p=245 Topic: Probing, Altering User Mode Memory This is part 3 of the intro to kernel debugging series.  Other posts: In this post, we will explore the following:
  • Probe memory of a user mode process
  • Alter user mode process memory
Reminders about how this tutorial is authored:
  • The author is a user mode developer, and tailors the conversation to that audience
  • Always make sure you have symbols loaded!
  • We are using a kd connection to a VM, as explained in a previous tutorial

Setup Test Apparatus

Let's create a simple user mode app to test with.  For simplicity, we want a test app that runs for a long time, making it easier to break into the app and inspect its state.  Here is the app we will be using: 
#include <windows.h>
#include <iostream>

void main(void)
{
    std::wcout << L"Hello, world" << std::endl;

    for (int i = 0; ; i++)
    {
        std::wcout << L"Loop iteration: " << i << std::endl;
        Sleep(1000);
    }
}
Having the test app wake up from time to time and inspect state will help illustrate the ability to modify state. In a previous tutorial, we used a kd connection to a local VM.  To set up the test apparatus, compile the above code and run the app on the VM OS.

Inspecting User Mode Process

Start by breaking into the debugger with control+break (windbg) or control+c (kd).  As always, make sure you have symbols loaded: 
Microsoft (R) Windows Debugger Version 10.0.10586.567 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Opened \\.\pipe\kd
Waiting to reconnect...
Connected to Windows 10 10240 x64 target at (Wed Jun 29 18:03:54.125 2016 (UTC - 7:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is: 
Windows 10 Kernel Version 10240 MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 10240.16841.amd64fre.th1_st1.160408-1853
Machine Name:
Kernel base = 0xfffff800`8f685000 PsLoadedModuleList = 0xfffff800`8f9aa070
Debug session time: Wed Jun 29 18:03:52.560 2016 (UTC - 7:00)
System Uptime: 0 days 4:34:42.939
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
*                                                                             *
*   You are seeing this message because you pressed either                    *
*       CTRL+C (if you run console kernel debugger) or,                       *
*       CTRL+BREAK (if you run GUI kernel debugger),                          *
*   on your debugger machine's keyboard.                                      *
*                                                                             *
*                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
*                                                                             *
* If you did not intend to break into the debugger, press the "g" key, then   *
* press the "Enter" key now.  This message might immediately reappear.  If it *
* does, press "g" and "Enter" again.                                          *
*                                                                             *
*******************************************************************************
nt!DbgBreakPointWithStatus:
fffff800`8f7d9bb0 cc              int     3
0: kd> .sympath cache*d:\sym;srv*
Symbol search path is: cache*d:\sym;srv*
Expanded Symbol search path is: cache*d:\sym;SRV*https://msdl.microsoft.com/download/symbols
************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       cache*d:\sym
Deferred                                       srv*
0: kd> .reload /f *.*
*** WARNING: Unable to verify timestamp for msrpc.sys
*** ERROR: Module load completed but symbols could not be loaded for msrpc.sys
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for clipsp.sys -
Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.
*** WARNING: Unable to verify timestamp for spaceport.sys
*** ERROR: Module load completed but symbols could not be loaded for spaceport.sys
*** WARNING: Unable to verify timestamp for volmgrx.sys
*** ERROR: Module load completed but symbols could not be loaded for volmgrx.sys
*** WARNING: Unable to verify timestamp for Fs_Rec.sys
*** ERROR: Module load completed but symbols could not be loaded for Fs_Rec.sys
*** WARNING: Unable to verify timestamp for Null.SYS
*** ERROR: Module load completed but symbols could not be loaded for Null.SYS
*** ERROR: Module load completed but symbols could not be loaded for peauth.sys
I gave a name to my test app, and I will be searching for that name in the process list: meason_test.exe.  The following command does the search: 
0: kd> !process 0 0 meason_test.exe
PROCESS ffffe0016723c840
    SessionId: 1  Cid: 1078    Peb: 7ff7e1ed5000  ParentCid: 0eec
    DirBase: 4a4ea000  ObjectTable: ffffc00063ac0c80  HandleCount: <Data Not Accessible>
    Image: meason_test.exe
Let's take a peek at what the process is doing.  We can do that by looking at the call stacks of all threads in the process, which can be done by passing the 0x17 value as the second argument to !process.  However, before we can do that, we must make sure the debugger can find the user mode symbols: 
0: kd> .sympath+ D:\_inc\test\amd64
Symbol search path is: cache*d:\sym;srv*;D:\_inc\test\amd64
Expanded Symbol search path is: cache*d:\sym;SRV*https://msdl.microsoft.com/download/symbols;d:\_inc\test\amd64
You may wish to optimize the symbol search by putting the private symbol path before the public symbol path, but we won't do that here. On to the process inspection with the 0x17 option.  Use the nt!_EPROCESS address that was output in the above !process search (it has the word "PROCESS" next to it in all caps). 
0: kd> !process ffffe0016723c840 17
PROCESS ffffe0016723c840
    SessionId: 1  Cid: 1078    Peb: 7ff7e1ed5000  ParentCid: 0eec
    DirBase: 4a4ea000  ObjectTable: ffffc00063ac0c80  HandleCount: <Data Not Accessible>
    Image: meason_test.exe
    VadRoot ffffe001667c91e0 Vads 20 Clone 0 Private 88. Modified 0. Locked 0.
    DeviceMap ffffc0006a2b8870
    Token                             ffffc00063c58060
    ElapsedTime                       00:00:31.376
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         18696
    QuotaPoolUsage[NonPagedPool]      2712
    Working Set Sizes (now,min,max)  (512, 50, 345) (2048KB, 200KB, 1380KB)
    PeakWorkingSetSize                486
    VirtualSize                       2097160 Mb
    PeakVirtualSize                   2097161 Mb
    PageFaultCount                    512
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      84

    Setting context for this process...                                       
            ffffffffffffffff  NotificationEvent
        Not impersonating
        DeviceMap                 ffffc0006a2b8870
        Owning Process            ffffe0016723c840       Image:         meason_test.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      1078268        Ticks: 4 (0:00:00:00.062)
        Context Switch Count      1265           IdealProcessor: 1             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.015
        Win32 Start Address meason_test!mainCRTStartup (0x00007ff7e281b9f0)
        Stack Init ffffd0010e766c90 Current ffffd0010e766790
        Base ffffd0010e767000 Limit ffffd0010e761000 Call 0
        Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

        Child-SP          RetAddr           : Args to Child                                                           : Call Site
        ffffd001`0e7667d0 fffff800`8f6d92a0 : ffffe001`00000000 00000000`00000001 00000000`00000000 ffffe001`63b4f440 : nt!KiSwapContext+0x76
        ffffd001`0e766910 fffff800`8f6d8cb8 : ffffe001`63b4f340 fffff800`8f9eb780 000000b6`b5522ee7 ffffe001`670bbf20 : nt!KiSwapThread+0x160
        ffffd001`0e7669c0 fffff800`8f709d89 : 00000027`3ac0b8cd 00000000`00000001 00000000`000000b0 000000fe`a11bf910 : nt!KiCommitThreadWait+0x148
        ffffd001`0e766a50 fffff800`8fb2ecac : ffffe001`63b4f340 00000000`00000000 00000000`00000000 000000fe`a11bf900 : nt!KeDelayExecutionThread+0x229
        ffffd001`0e766ad0 fffff800`8f7deb63 : ffffe001`63b4f340 00007ff7`e1ed5000 ffffffff`ff676980 ffffe001`6427c350 : nt!NtDelayExecution+0x5c
        ffffd001`0e766b00 00007fff`c5803b6a : 00007fff`c2be3757 000000fe`a11bfbd8 000000fe`a11bfb01 ffffffff`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd001`0e766b00)
        000000fe`a11bfb18 00007fff`c2be3757 : 000000fe`a11bfbd8 000000fe`a11bfb01 ffffffff`00000000 00007ff7`e2821440 : ntdll!NtDelayExecution+0xa
        000000fe`a11bfb20 00007ff7`e2818323 : 00000000`00000001 00007ff7`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!SleepEx+0xa7
        000000fe`a11bfbc0 00007ff7`e281b96d : 00000000`00000001 00000000`00000000 00000000`00000000 00007ff7`e281d3d0 : meason_test!main+0x73 [d:\test\meason_test\main.cpp @ 13]
        000000fe`a11bfc00 00007fff`c3c22d92 : 00007ff7`e281b9f0 00007ff7`e1ed5000 00007ff7`e1ed5000 00000000`00000000 : meason_test!__mainCRTStartup+0x14d [d:\(omitted) @ 697]
        000000fe`a11bfc40 00007fff`c5779f64 : 00007fff`c3c22d70 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x22
        000000fe`a11bfc70 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x34

Probe User Mode Memory

At this point, we could attempt to query user mode memory with the 'u' command (unassemble).  However, it likely won't work as expected.  Let's inspect a return address from the call stack above (chosen carefully to ensure we are looking at our own app): 
0: kd> u 00007ff7`e2818323
00007ff7`e2818323 ??              ???
                         ^ Memory access error in 'u 00007ff7`e2818323'
As we learned in the last tutorial, when you first break in with the kernel debugger, your debugger context may not necessarily be set to where you want it to be.  That's why we are unable to read this address: the debugger is using some other process context, and it is one in which this address does not translate to a valid page. Let's set the right debugger context for our test app.  Use the nt!_EPROCESS address that was output in the !process command above. 
0: kd> .process /p /r ffffe0016723c840
Implicit process is now ffffe001`6723c840
.cache forcedecodeuser done
Loading User Symbols
.....

************* Symbol Loading Error Summary **************
Module name            Error
msrpc                  The system cannot find the file specified
clipsp                 The system cannot find the file specified
spaceport              The system cannot find the file specified
volmgrx                The system cannot find the file specified
Fs_Rec                 The system cannot find the file specified
Null                   The system cannot find the file specified
peauth                 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
You should now be able to unassemble the user mode call stack address: 
0: kd> ub 00007ff7`e2818323
meason_test!main+0x4d [d:\test\meason_test\main.cpp @ 11]:
00007ff7`e28182fd 8b542420        mov     edx,dword ptr [rsp+20h]
00007ff7`e2818301 488bc8          mov     rcx,rax
00007ff7`e2818304 e887a6ffff      call    meason_test!std::basic_ostream<wchar_t,std::char_traits<wchar_t> >::operator<< (00007ff7`e2812990)
00007ff7`e2818309 488d15a0f1ffff  lea     rdx,[meason_test!std::endl (00007ff7`e28174b0)]
00007ff7`e2818310 488bc8          mov     rcx,rax
00007ff7`e2818313 e8e8a8ffff      call    meason_test!std::basic_ostream<wchar_t,std::char_traits<wchar_t> >::operator<< (00007ff7`e2812c00)
00007ff7`e2818318 b9e8030000      mov     ecx,3E8h
00007ff7`e281831d ff159d4d0000    call    qword ptr [meason_test!_imp_Sleep (00007ff7`e281d0c0)]
0: kd> u
meason_test!main+0x73 [d:\test\meason_test\main.cpp @ 13]:
00007ff7`e2818323 ebbb            jmp     meason_test!main+0x30 (00007ff7`e28182e0)
00007ff7`e2818325 4883c438        add     rsp,38h
00007ff7`e2818329 c3              ret
00007ff7`e281832a cc              int     3
00007ff7`e281832b cc              int     3
00007ff7`e281832c cc              int     3
00007ff7`e281832d cc              int     3
00007ff7`e281832e cc              int     3
Observe that we have user mode code showing up in the kernel debugger!  The use of the 'ub' unassembled command was chosen in order to show the assembly code leading up to our Sleep() call.

Alter User Memory

Let's start messing around with the memory in our app!  Recall that our test app from above had a running counter.  Let's edit its value.  While the app is running (and the kernel debugger is in run mode, not break mode), you will see the loop iteration keeps increasing: 
c:\test\amd64>meason_test.exe
Hello, world
Loop iteration: 0
Loop iteration: 1
Loop iteration: 2
Loop iteration: 3
Loop iteration: 4
Loop iteration: 5
Break into the debugger, find our test process, and set the debugger context to that process: 
0: kd> !process 0 0 meason_test.exe
PROCESS ffffe00138301840
    SessionId: 1  Cid: 0450    Peb: 7ff7a568f000  ParentCid: 0cb0
    DirBase: 21cb6000  ObjectTable: ffffc001ae412840  HandleCount: <Data Not Accessible>
    Image: meason_test.exe
0: kd> .process /p /r ffffe00138301840
Implicit process is now ffffe001`38301840
.cache forcedecodeuser done
Loading User Symbols
.....
Using !process, let's iterate over all threads in the process:
0: kd> !process ffffe00138301840 f
PROCESS ffffe00138301840
    SessionId: 1  Cid: 0450    Peb: 7ff7a568f000  ParentCid: 0cb0
    DirBase: 21cb6000  ObjectTable: ffffc001ae412840  HandleCount: <Data Not Accessible>
    Image: meason_test.exe
    VadRoot ffffe00138d16010 Vads 20 Clone 0 Private 89. Modified 0. Locked 0.
    DeviceMap ffffc001a6c4c710
    Token                             ffffc001a7633060
    ElapsedTime                       00:01:28.220
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         18696
    QuotaPoolUsage[NonPagedPool]      2712
    Working Set Sizes (now,min,max)  (518, 50, 345) (2072KB, 200KB, 1380KB)
    PeakWorkingSetSize                491
    VirtualSize                       2097160 Mb
    PeakVirtualSize                   2097162 Mb
    PageFaultCount                    519
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      86

        THREAD ffffe00138d9a640  Cid 0450.0e2c  Teb: 00007ff7a568d000 Win32Thread: 0000000000000000 WAIT: (DelayExecution) UserMode Non-Alertable
            ffffffffffffffff  NotificationEvent
        Not impersonating
        DeviceMap                 ffffc001a6c4c710
        Owning Process            ffffe00138301840       Image:         meason_test.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      11054          Ticks: 64 (0:00:00:01.000)
        Context Switch Count      1114           IdealProcessor: 1             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.031
        Win32 Start Address meason_test!mainCRTStartup (0x00007ff7a5b9b9f0)
        Stack Init ffffd0015490dc90 Current ffffd0015490d790
        Base ffffd0015490e000 Limit ffffd00154908000 Call 0
        Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 

        Child-SP          RetAddr           Call Site
        ffffd001`5490d7d0 fffff801`8f4592a0 nt!KiSwapContext+0x76
        ffffd001`5490d910 fffff801`8f458cb8 nt!KiSwapThread+0x160
        ffffd001`5490d9c0 fffff801`8f489d89 nt!KiCommitThreadWait+0x148
        ffffd001`5490da50 fffff801`8f8aecac nt!KeDelayExecutionThread+0x229
        ffffd001`5490dad0 fffff801`8f55eb63 nt!NtDelayExecution+0x5c
        ffffd001`5490db00 00007ffa`5a313b6a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd001`5490db00)
        000000c5`afc5f7d8 00007ffa`57743757 ntdll!NtDelayExecution+0xa
        000000c5`afc5f7e0 00007ff7`a5b98323 KERNELBASE!SleepEx+0xa7
        000000c5`afc5f880 00007ff7`a5b9b96d meason_test!main+0x73 [d:\test\meason_test\main.cpp @ 13]
        000000c5`afc5f8c0 00007ffa`57f42d92 meason_test!__mainCRTStartup+0x14d [(omitted) @ 697]
        000000c5`afc5f900 00007ffa`5a289f64 KERNEL32!BaseThreadInitThunk+0x22
        000000c5`afc5f930 00000000`00000000 ntdll!RtlUserThreadStart+0x34
Our process only has one thread, so choosing the relevant thread here is trivial.  Set the debugger context to that thread, so we can pull up its call stack:
0: kd> .thread /p /r ffffe00138d9a640  
Implicit thread is now ffffe001`38d9a640
Implicit process is now ffffe001`38301840
.cache forcedecodeuser done
Loading User Symbols
.....
0: kd> k
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 ffffd001`5490d7d0 fffff801`8f4592a0 nt!KiSwapContext+0x76
01 ffffd001`5490d910 fffff801`8f458cb8 nt!KiSwapThread+0x160
02 ffffd001`5490d9c0 fffff801`8f489d89 nt!KiCommitThreadWait+0x148
03 ffffd001`5490da50 fffff801`8f8aecac nt!KeDelayExecutionThread+0x229
04 ffffd001`5490dad0 fffff801`8f55eb63 nt!NtDelayExecution+0x5c
05 ffffd001`5490db00 00007ffa`5a313b6a nt!KiSystemServiceCopyEnd+0x13
06 000000c5`afc5f7d8 00007ffa`57743757 ntdll!NtDelayExecution+0xa
07 000000c5`afc5f7e0 00007ff7`a5b98323 KERNELBASE!SleepEx+0xa7
08 000000c5`afc5f880 00007ff7`a5b9b96d meason_test!main+0x73 [d:\test\meason_test\main.cpp @ 13]
09 000000c5`afc5f8c0 00007ffa`57f42d92 meason_test!__mainCRTStartup+0x14d [(omitted) @ 697]
0a 000000c5`afc5f900 00007ffa`5a289f64 KERNEL32!BaseThreadInitThunk+0x22
0b 000000c5`afc5f930 00000000`00000000 ntdll!RtlUserThreadStart+0x34
We are going to alter the loop iteration variable, which is a variable stored on the stack.  It is located in the main() function, so let's switch debugger context to that frame of the call stack.  The frame number is 8, which we can see as the first column in the 'k' output above.
0: kd> .frame 8
08 000000c5`afc5f880 00007ff7`a5b9b96d meason_test!main+0x73 [d:\test\meason_test\main.cpp @ 13]
0: kd> dv
              i = 0n27
How do you find the address of this stack variable?  There are many ways, but we'll use the most clunky method: Look at disassembly.  Let's look for the variable being incremented in the main function:
0: kd> u meason_test!main
meason_test!main [d:\test\meason_test\main.cpp @ 6]:
00007ff7`a5b982b0 4883ec38        sub     rsp,38h
00007ff7`a5b982b4 488d1515580000  lea     rdx,[meason_test!`string' (00007ff7`a5b9dad0)]
00007ff7`a5b982bb 488d0d7e8d0000  lea     rcx,[meason_test!std::wcout (00007ff7`a5ba1040)]
00007ff7`a5b982c2 e8d98effff      call    meason_test!std::operator<<<wchar_t,std::char_traits<wchar_t> > (00007ff7`a5b911a0)
00007ff7`a5b982c7 488d15e2f1ffff  lea     rdx,[meason_test!std::endl (00007ff7`a5b974b0)]
00007ff7`a5b982ce 488bc8          mov     rcx,rax
00007ff7`a5b982d1 e82aa9ffff      call    meason_test!std::basic_ostream<wchar_t,std::char_traits<wchar_t> >::operator<< (00007ff7`a5b92c00)
00007ff7`a5b982d6 c744242000000000 mov     dword ptr [rsp+20h],0
0: kd> u
meason_test!main+0x2e [d:\test\meason_test\main.cpp @ 9]:
00007ff7`a5b982de eb0a            jmp     meason_test!main+0x3a (00007ff7`a5b982ea)
00007ff7`a5b982e0 8b442420        mov     eax,dword ptr [rsp+20h]
00007ff7`a5b982e4 ffc0            inc     eax
00007ff7`a5b982e6 89442420        mov     dword ptr [rsp+20h],eax
00007ff7`a5b982ea 488d15ff570000  lea     rdx,[meason_test!`string' (00007ff7`a5b9daf0)]
00007ff7`a5b982f1 488d0d488d0000  lea     rcx,[meason_test!std::wcout (00007ff7`a5ba1040)]
00007ff7`a5b982f8 e8a38effff      call    meason_test!std::operator<<<wchar_t,std::char_traits<wchar_t> > (00007ff7`a5b911a0)
00007ff7`a5b982fd 8b542420        mov     edx,dword ptr [rsp+20h]
The 'inc eax' instruction is the one we're looking for.  As we can see in the instruction above the 'inc eax' instruction, the EAX register gets loaded from whatever is at address [rsp+20h].  That is, 0x20 offset from the RSP register for the frame.  We can't rely on the actual value in the RSP register anymore,  as it is now loaded with data that is relative to frame 0 of our call stack.  Using the 'k' output from above, we see the stack pointer for this frame is 000000c5`afc5f880.  Thus, we retrieve data at 0x20 offset from that address:
0: kd> dd 000000c5`afc5f880+0x20
000000c5`afc5f8a0  0000001b 00000000 00000000 00000000
000000c5`afc5f8b0  00000000 00000000 a5b9b96d 00007ff7
000000c5`afc5f8c0  00000001 00000000 00000000 00000000
000000c5`afc5f8d0  00000000 00000000 a5b9d3d0 00007ff7
000000c5`afc5f8e0  00000000 00000000 a5b9d3d0 00007ff7
000000c5`afc5f8f0  00000000 00000000 57f42d92 00007ffa
000000c5`afc5f900  a5b9b9f0 00007ff7 a568f000 00007ff7
000000c5`afc5f910  a568f000 00007ff7 00000000 00000000
0: kd> ? 0000001b
Evaluate expression: 27 = 00000000`0000001b
 Notice how we used the 'dd' debugger command to dump 32-bit values, as our code is using a 32-bit data type.  As we can see, the loop iteration is at number 27 in our program's execution.
Let's alter the value to something wacky, just to be sure we edited the value we want.  Use the 'ed' debugger command to edit the value at the address of this stack variable.  Notice that this only works when the debugger context is set to this process, so the debugger knows how to translate this address properly.
0: kd> ed 000000c5`afc5f880+0x20 0x12345678
0: kd> dd 000000c5`afc5f880+0x20
000000c5`afc5f8a0  12345678 00000000 00000000 00000000
000000c5`afc5f8b0  00000000 00000000 a5b9b96d 00007ff7
000000c5`afc5f8c0  00000001 00000000 00000000 00000000
000000c5`afc5f8d0  00000000 00000000 a5b9d3d0 00007ff7
000000c5`afc5f8e0  00000000 00000000 a5b9d3d0 00007ff7
000000c5`afc5f8f0  00000000 00000000 57f42d92 00007ffa
000000c5`afc5f900  a5b9b9f0 00007ff7 a568f000 00007ff7
000000c5`afc5f910  a568f000 00007ff7 00000000 00000000
Now, let's run the app and see what the result looks like.  Make sure to resume the debugger so the OS can start executing again.
Loop iteration: 25
Loop iteration: 26
Loop iteration: 27
Loop iteration: 305419897
Loop iteration: 305419898
Loop iteration: 305419899
That first large value corresponds to 0x12345679, which is what we would expect after coming out of the Sleep() point of the app and starting on the next loop iteration.

Summary

In this tutorial, we probed user memory with a sample app.  We then altered the memory and saw the result.  In all cases, the debugger context had to be set appropriately, else nothing would happen.
]]> Intro to kernel debugging 1 https://docs.microsoft.com/archive/blogs/meason/intro-to-kernel-debugging-1 Fri, 17 Jun 2016 21:13:13 GMT https://blogs.msdn.microsoft.com/meason/?p=215 Topic: KD Setup I am a user-mode developer, but part of the job of working on the Windows team (HoloLens runs on Windows!) requires knowing how to work with a kernel debugger on that OS.  Some problems are difficult to debug through user-mode debuggers alone and can be simpler in a kernel debugger . Examples include:
  • Failure to launch process
  • Early-stage OS boot or similar
  • Inter-process communication
But how does one learn how to use the kernel debugger on Windows if the code you write only runs in user mode?  Many tutorials are intended for driver authors.  I intend to author a brief intro to kernel debugging from the perspective of someone who doesn't write code there.  However, my perspective also includes being a Microsoft employee.  As such, I have access to source code and symbols that the general public does not have.  Kernel debugging is likely more applicable to someone in my position. There are some topics that you should learn outside of this tutorial that will make you more effective as a kernel debugger:
  • Familiarity with debugging, particularly with any one of: {windbg, cdb, kd}
  • Difference between kernel mode and user mode execution
  • High-level understanding of interrupts and IRQLs
I learned about these topics while on the job or through reading the "Windows Internals" book by Russinovich / Solomon / Ionescu.  If you don't want to read that book, find some other way to get familiar with these concepts, as it really helps. Let's get started with kd! Note: This tutorial is part of a series.  See other parts of the series here:
 

Terminology

  • Debug target - The machine being interrogated by the debugger
  • Debug host - The machine doing the interrogating through the debugger.  Likely your dev machine.

Set up apparatus

In the past, setting up a kd was a cumbersome activity.  Today, we can whip up a virtual machine and hook up a kernel debugger with a few commands.  The following is how a kd is set up through a Hyper-V based machine:
  1. Install your Windows OS on the VM
  2. In the Hyper-V settings for the VM, set COM 1 to use a Named Pipe.  I will name mine "test1", which ends up creating a named pipe at \\.\pipe\test1
  3. In an elevated command prompt from within the OS running in the VM (the "debug target"), execute the following commands:
    1. bcdedit /set {default} debug on
    2. bcdedit /dbgsettings serial debugport:1 baudrate:115200
  4. Launch your favorite kernel debugger from your debug host (your dev machine).  My favorite is windbg.exe.
    1. (In windbg, I press control+K to open up the kd window and specify settings through there)
    2. Set the debugger to connect to connect with the following settings:
      1. Type = COM
      2. Baud Rate = 115200
      3. Connect through Pipe
      4. Port = \\.\pipe\test1
      5. Set it to reconnect automatically with Resets = 0
  5. Reboot the VM OS (the "debug target")
  6. As soon as the debug target gets far enough along in the boot process, the kernel debugger will automatically attach
You will see something similar to the following when the debugger officially attaches: 
Opened \\.\pipe\test1
Waiting to reconnect...
Connected to Windows 10 xxxxx x86 compatible target at (Fri Jun 10 14:26:44.374 2016 (UTC - 7:00)), ptr64 FALSE
Kernel Debugger connection established.

First kernel debugging commands

After you have connected, you can break in at any moment in order to see what's going on.  Press control+break (windbg) or control+c (kd, cdb) to break in: 
Break instruction exception - code 80000003 (first chance)
  *******************************************************************************
  *                                                                             *
  *   You are seeing this message because you pressed either                    *
  *       CTRL+C (if you run console kernel debugger) or,                       *
  *       CTRL+BREAK (if you run GUI kernel debugger),                          *
  *   on your debugger machine's keyboard.                                      *
  *                                                                             *
  *                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
  *                                                                             *
  * If you did not intend to break into the debugger, press the "g" key, then   *
  * press the "Enter" key now.  This message might immediately reappear.  If it *
  * does, press "g" and "Enter" again.                                          *
  *                                                                             *
  *******************************************************************************
  nt!RtlpBreakWithStatusInstruction:
  819a4bc4 cc              int     3
The first thing I always do when connecting to a debugger is make sure the symbols are resolved, loaded, and cached.  I normally have my sympath set in the _NT_SYMBOL_PATH environment variable, but you can also set it explicitly with the ".sympath" command.  Non-Microsoft employees should include the public symbol server at Microsoft as follows: 
2: kd> .sympath cache*c:\sym;c:\MySymbolPath1;\\MySymbolServer\SomePath2\foo\bar;srv*https://msdl.microsoft.com/download/symbols
 Symbol search path is: cache*c:\sym;c:\MySymbolPath1;\\MySymbolServer\SomePath2\foo\bar;srv*https://msdl.microsoft.com/download/symbols
 Expanded Symbol search path is: cache*c:\sym;c:\mysymbolpath1;\\mysymbolserver\somepath2\foo\bar;srv*https://msdl.microsoft.com/download/symbols
************* Symbol Path validation summary **************
 Response                         Time (ms)     Location
 Deferred                                       cache*c:\sym
 Deferred                                       c:\MySymbolPath1
 Deferred                                       \\MySymbolServer\SomePath2\foo\bar
 Deferred                                       srv*https://msdl.microsoft.com/download/symbols
Once the sympath is set, try loading all symbols in order to get the major symbols cached locally.  If this is your first time doing this, it can take a long time, on the order of 3-5 minutes.  Symbol files can be large. 
2: kd> .reload /f *.*
Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.
If a particular file has trouble loading and you feel you need it, turn on verbose symbol resolving (!sym noisy) and force that one module to reload (.reload /f foo.dll). Once symbols have loaded, you can use the 'lm' command to see which symbols (if any) are loaded for each module: 
2: kd> lm
start    end        module name
81341000 8134c000   kdcom      (private pdb symbols)  c:\sym\kdcom.pdb\8F834143CF1A42A3BC536396DA9853A91\kdcom.pdb
8181d000 8187e000   hal        (private pdb symbols)  c:\sym\halmacpi.pdb\EE1E389525C24DBEBF46DDDD10F6F6DF1\halmacpi.pdb
8187e000 81e95000   nt         (private pdb symbols)  c:\sym\ntkrpamp.pdb\73A615FD399C45A7A669ACFBAEBC82201\ntkrpamp.pdb
82000000 82072000   storport   (private pdb symbols)  c:\sym\storport.pdb73EE2F1FFD246009F9E36C45650BBF91\storport.pdb
82080000 8208a000   Fs_Rec     (private pdb symbols)  c:\sym\fs_rec.pdb\765EE27940FF458D95602B9CC799F83D1\fs_rec.pdb
82090000 820b7000   ksecpkg    (private pdb symbols)  c:\sym\ksecpkg.pdb\E5C81256D7F84A03B25B69E203B2A8261\ksecpkg.pdb
820c0000 820d1000   mpsdrv     (private pdb symbols)  c:\sym\mpsdrv.pdb\37DB180E8ABE4217B848B65D56E0B8481\mpsdrv.pdb
820f0000 82106000   mountmgr   (private pdb symbols)  c:\sym\mountmgr.pdb\9F665C8CB77344BE8D230633D672E6C71\mountmgr.pdb
82110000 82117000   intelide   (private pdb symbols)  c:\sym\intelide.pdb\A0F94CA95D964E47A325BA72DF9CCFF91\intelide.pdb
82120000 8212e000   PCIIDEX    (private pdb symbols)  c:\sym\pciidex.pdb\27855013369A402FA201B2DAFD92DE1B1\pciidex.pdb
82130000 82139000   atapi      (private pdb symbols)  c:\sym\atapi.pdb\CC79A012F9464F3BAA9D1B3926F6275D1\atapi.pdb
82140000 82169000   ataport    (private pdb symbols)  c:\sym\ataport.pdb\2274ED5AC2B344A8807C0CDD155B154A1\ataport.pdb
82170000 82182000   fileinfo   (private pdb symbols)  c:\sym\fileinfo.pdb\81692D1241464AE898EF26401F6FD0D11\fileinfo.pdb
82190000 821a2000   WimFsf     (private pdb symbols)  c:\sym\wimfsf.pdb\85AD06671DFF4CCABC15341F12D5571C1\wimfsf.pdb
821b0000 82200000   ks         (private pdb symbols)  c:\sym\ks.pdb\432B6F981F0441EDBED8C24DA0C4C2151\ks.pdb
82400000 825da000   Ntfs       (private pdb symbols)  c:\sym\ntfs.pdb\E9EF50EFE34D41FEA4703C0E78C9B0921\ntfs.pdb
825e0000 825ea000   storvsc    (private pdb symbols)  c:\sym\storvsc.pdb\BDCC5C0D12B44942AFB8CC7339081E401\st
(truncated for brevity)
Having symbols loaded for some critical modules is vital to making any sense of anything.  If you aren't getting symbols to load for the 'nt' module, stop what you are doing and figure it out.  Most debugger commands will likely NOT work properly unless you have the 'nt' symbols loaded.  Most other modules in the kernel space are safe to not have symbols loaded for ordinary debugging, but 'nt' is crucial.

Exploring kernel land

My key to exploring the kernel space is the !process command.  I recommend looking at your debugger docs for the command; the debugger docs are well-maintained and very informative.  The !process command allows you to enumerate and query data for all processes in the system.  Here are some sample uses for !process:

Enumerate all processes

2: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 9fc41040  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 001a9000  ObjectTable: 82804000  HandleCount: 510.
    Image: System
PROCESS a514c8c0  SessionId: none  Cid: 0174    Peb: 02471000  ParentCid: 0004
    DirBase: 7ffe0020  ObjectTable: 829a8bc0  HandleCount:  46.
    Image: smss.exe
(truncated for brevity)
 

Locate a specific process

This example is looking for all instances of meason_test.exe, which is a test app I created that does nothing but Sleep(INFINITE). 
1: kd> !process 0 0 meason_test.exe
PROCESS ab870bc0  SessionId: 0  Cid: 01dc    Peb: 00458000  ParentCid: 0af8
    DirBase: 7ffe04c0  ObjectTable: b0533040  HandleCount:  28.
    Image: meason_test.exe
 

Query all threads for a specific process

This example references the PROCESS address found in the previous example, in order to restrict output to one specific process (rather than all processes that match the string specified) 
1: kd> !process ab870bc0 2
PROCESS ab870bc0  SessionId: 0  Cid: 01dc    Peb: 00458000  ParentCid: 0af8
    DirBase: 7ffe04c0  ObjectTable: b0533040  HandleCount:  28.
    Image: meason_test.exe
        THREAD ac5435c0  Cid 01dc.0fe8  Teb: 00459000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Non-Alertable
            ffffffff  NotificationEvent
        THREAD ae63ba00  Cid 01dc.095c  Teb: 0045a000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable
            ae6147c0  QueueObject
        THREAD 9fcac040  Cid 01dc.07cc  Teb: 0045b000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable
            ae6147c0  QueueObject

 Query call stacks for all threads

This command lets you see the kernel mode portion of the call stack for each thread.  Note, I had to obscure the nt and ntdll symbol names and source code paths, as I am not sure what the private symbol server exposes relative to the public symbol server. 
1: kd> !process ab870bc0 17
PROCESS ab870bc0  SessionId: 0  Cid: 01dc    Peb: 00458000  ParentCid: 0af8
    DirBase: 7ffe04c0  ObjectTable: b0533040  HandleCount:  28.
    Image: meason_test.exe
    VadRoot ab89f730 Vads  Clone 0 Private . Modified . Locked .
    DeviceMap 82807ba0
    Token                             ad57c438
    ElapsedTime                       00:00:04.735
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         16472
    QuotaPoolUsage[NonPagedPool]      1472
    Working Set Sizes (now,min,max)  (, , ) (KB, KB, KB)
    PeakWorkingSetSize                
    VirtualSize                        Mb
    PeakVirtualSize                    Mb
    PageFaultCount                    
    MemoryPriority                    BACKGROUND
    BasePriority                      
    CommitCharge                     
        THREAD ac5435c0  Cid 01dc.0fe8  Teb: 00459000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Non-Alertable
            ffffffff  NotificationEvent
        Not impersonating
        DeviceMap                 82807ba0
        Owning Process                   Image:         meason_test.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      1386973        Ticks:  (0:00:00:04.015)
        Context Switch Count      39             IdealProcessor: 3             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address meason_test!mainCRTStartup (0x00ad5bc0)
        Stack Init  Current  Base  Limit  Call 
        Priority  BasePriority  PriorityDecrement  IoPriority 2 PagePriority 5

        ChildEBP RetAddr  Args to Child              
        a9b7cc04 818be3a5 00000000 ac543678 ac5435c0 nt!(omitted)+0x19 (FPO: [Uses EBP] [1,0,4]) [(omitted)]
        a9b7cc78 818bde89 ac5435c0 a9b7cd1c 81af9101 nt!(omitted)+0x195 (FPO: [Non-Fpo]) (CONV: fastcall) [(omitted)]
        a9b7cccc 818b495d 00000002 7531db3b 80000032 nt!(omitted)+0x159 (FPO: [Non-Fpo]) (CONV: stdcall) [(omitted)]
        a9b7ccf8 81af9189 ffffff00 00000000 00000000 nt!(omitted)+0xad (FPO: [Non-Fpo]) (CONV: stdcall) [(omitted)]
        a9b7cd44 819b2097 00000000 003af8b8 003af8dc nt!(omitted)+0x89 (FPO: [Non-Fpo]) (CONV: stdcall) [(omitted)]
        a9b7cd44 77302090 00000000 003af8b8 003af8dc nt!(omitted) (FPO: [0,3] TrapFrame @ a9b7cd54) [(omitted)]
        003af870 77300bea 76fae258 00000000 003af8b8 ntdll!(omitted) (FPO: [0,0,0]) [(omitted)]
        003af874 76fae258 00000000 003af8b8 0ebd6750 ntdll!(omitted) +0xa (FPO: [2,0,0]) [(omitted)]
        003af8dc 76fae1af ffffffff 00000000 003af8f8 KERNELBASE!SleepEx+0x98 (FPO: [SEH]) (CONV: stdcall) [(omitted)]
        003af8ec 00ad2c38 ffffffff 003af93c 00ad5aff KERNELBASE!Sleep+0xf (FPO: [Non-Fpo]) (CONV: stdcall) [(omitted)]
        003af8f8 00ad5aff 00000001 009e1ed8 009e1230 meason_test!main+0x38 (FPO: [Non-Fpo]) (CONV: cdecl) [(omitted)]
        003af93c 77291154 00458000 744ffd4f 00000000 meason_test!__mainCRTStartup+0x107 (FPO: [Non-Fpo]) (CONV: cdecl) [(omitted)]
        003af984 77291114 ffffffff 7731448a 00000000 ntdll(omitted)+0x3a (FPO: [SEH]) (CONV: stdcall) [(omitted)]
        003af994 00000000 00ad5bc0 00458000 00000000 ntdll(omitted)+0x1b (FPO: [Non-Fpo]) (CONV: stdcall) [(omitted)]
(Truncated for brevity)
 

Summary

In this tutorial, we didn't get very deep into the bowels of the OS.  However, we cracked open the door and took a peek.  In the next tutorial, we will get a peek at what the kernel debugger looks like when it first breaks in.]]> Intro to kernel debugging 2 https://docs.microsoft.com/archive/blogs/meason/intro-to-kernel-debugging-2 Fri, 17 Jun 2016 16:42:11 GMT https://blogs.msdn.microsoft.com/meason/?p=225 Topic: Debugger Context This is part 2 of the intro to kernel debugging series.  Other posts: In this post, we will explore the following:
  • What the debugger is looking at when it first breaks in
  • Get current call stack
  • Get current process
  • Get current processor
  • Get current IRQL
Reminders about how this tutorial is authored:
  • The author is a user mode developer, and tailors the conversation to that audience
  • Always make sure you have symbols loaded!
  • We are using a kd connection to a VM, as explained in a previous tutorial
 

Context of debugger break-in

When your debugger has broken into the target, you may notice that the target operating system has become completely unresponsive.  That's what kernel debugging is: You're debugging the operating system itself, and it needs to be frozen in order to inspect it. With your operating system frozen, what you are looking at in the debugger is the state of the system when it became frozen.  That includes all processors, all processes, all threads in all processes, and even the operating systems threads.  The debugger will be looking at something relative to where the break-in occurred (its context).  Some potential debugger break-in points include:
  • Kernel breakpoint
    • Debugger breakpoint for kernel mode code
  • User breakpoint
    • Debugger breakpoint for user mode code
  • Bugcheck
    • The infamous bluescreen / BSOD
  • DbgBreakpoint()
    • Code-defined breakpoint
  • Clock interrupt check for break-in request
    • User pressed break keystroke on debugger, and the OS clock interrupt monitors for the request
  Generally, there are up to 4 items that developers immediately want to know about to establish context once the debugger has broken in:
  1. Call stack
  2. Current process
  3. Current processor
  4. Current IRQL
In order to establish context, it seems all developers are interested in #1: What's the call stack?  However, there is a divergence with the others.  User mode developers tend to care more about #2, whereas kernel developers tend to care more about #3 and #4.  We will cover each.  

Get call stack

If you've been using {windbg, cdb, kd} for a while, you should already know how to do this.  If not, take a look at the documentation for it.  The debugger docs are well-maintained, and are replicated to MSDN. The most basic call stack is available with the 'k' command.  Note that the output here has been modified to obscure some data, as I am not sure what is exposed to the public versus what is exposed to Microsoft employees: 
1: kd> k
# ChildEBP RetAddr
00 9f1e69f8 819fa3a6 nt!(omitted) [(omitted) @ 48]
01 9f1e6a1c 8191a4cd nt!(omitted)+0x74a96 [(omitted) @ 244]
02 9f1e6a40 8191916b nt!(omitted)+0x3d [(omitted) @ 672]
03 9f1e6a8c 81825e70 nt!(omitted)+0x10b [(omitted) @ 1331]
04 9f1e6a90 81837857 hal!(omitted)+0x6 [(omitted) @ 328]
05 9f1e6a90 8183941b hal!(omitted)+0x1f7 [(omitted) @ 228]
06 9f1e6b94 81a4866d hal!(omitted)+0x7 [(omitted) @ 87]
07 9f1e6ba4 8191be83 nt!(omitted)+0x31 [(omitted) @ 671]
08 9f1e6cc8 8191b836 nt!(omitted)+0x513 [(omitted) @ 3856]
09 9f1e6d44 819b69a1 nt!(omitted)+0x306 [(omitted) @ 1046]
0a 9f1e6d48 00000000 nt!KiIdleLoop+0xd [(omitted) @ 1451]
Many people use exotic versions of the 'k' command.  Read the docs and come up with your own.  The author prefers 'kpnL', which removes source code data and inserts param names and values.  

Get current process

When the kd breaks in, you'll always be in the context of some process running on the CPU serving up your debugger break (even threads executing in the system space are part of a process).  For user mode processes, a lot of the CPU state will refer to virtual addresses relative to that process.  Hence, it becomes vital to ensure the debugger context is set to the desired process so that a proper translation of virtual addresses occurs. To query the current process context that the debugger is using, execute !process -1 0: 
2: kd> !process -1 0
PROCESS a027cb80  SessionId: 0  Cid: 02b0    Peb: 02f7b000  ParentCid: 0210
    DirBase: 7ffe0520  ObjectTable: af387300  HandleCount: 694.
    Image: dwm.exe
If you then use the 'k' command to enumerate the call stack, you'll be seeing the stack for a thread in this process (dwm.exe in the example shown here).  Address translation and symbol lookups will be taken care of appropriately by the debugger. While !process allows you to view the state of any process, this command will not set a new debugger context.  There is one exception to this rule, when you add the appropriate bit (0x10) to your display options.  For example, with flags 0x17 set, it will temporarily switch context for the purpose of output in !process.  To set a new process context, use the .process command.  User mode developers will want to add the /p /r options to enable automatic address translations in the context of this new process. 
3: kd> !process -1 0
PROCESS a027cb80  SessionId: 0  Cid: 02b0    Peb: 02f7b000  ParentCid: 0210
    DirBase: 7ffe0520  ObjectTable: af387300  HandleCount: 694.
    Image: dwm.exe
3: kd> .process
Implicit process is now a027cb80
3: kd> !process 0 0 meason_test.exe
PROCESS acac1040  SessionId: 0  Cid: 0bd8    Peb: 0290d000  ParentCid: 0f40
    DirBase: 7ffe05a0  ObjectTable: b2248580  HandleCount:  28.
    Image: meason_test.exe
3: kd> .process /p /r acac1040
Implicit process is now acac1040
.cache forcedecodeuser done
Loading User Symbols
....
Note that this sequence of commands utilized !process to perform a process search. What can we do now that the process context is set to a particular process?  You can probe all threads and get user mode call stacks for each.  Address translation is done for you by the debugger.  If you attempt to output a user mode call stack while the process context is not set to that process, you will get no symbols at best!  

Get current processor, IRQL

The kd breakin must occur on one processor in your multiprocessor system, and the CPU is running at a particular IRQL when the break occurs on the processor.  Sometimes, you may need to reference these values. The current processor is shown directly in your kd prompt.  For example, the following is within the context of CPU #2: 
2: kd>
In {cdb, kd}, it is readily apparent as part of the currently executing command.  In {windbg}, it is shown in the graphical interface next to where you enter in commands. At the time of the breakin, the CPU executing the breakin code was running at a particular IRQL.  You can view this IRQL via the !irql command, which also displays the current CPU: 
2: kd> !irql
Debugger saved IRQL for processor 0x2 -- 28 (CLOCK2_LEVEL)
From the example above, I can see it was running at a really high IRQL - the clock interrupt.  This generally means the debugger user manually requested a break in.  You will see more interesting data with breakpoints or bugchecks. You can switch debugger context to other processors by executing the context switch command (~s): 
2: kd> !irql
Debugger saved IRQL for processor 0x2 -- 28 (CLOCK2_LEVEL)
2: kd> ~0s
0: kd> !irql
Debugger saved IRQL for processor 0x0 -- 0 (LOW_LEVEL)
0: kd> ~1s
1: kd> !irql
Debugger saved IRQL for processor 0x1 -- 0 (LOW_LEVEL)
1: kd> ~3s
3: kd> !irql
Debugger saved IRQL for processor 0x3 -- 0 (LOW_LEVEL)
 

Summary

In this tutorial, we get to see what the kernel debugger looks like when it first breaks in.  We also got to migrate around a little bit to view various artifacts of the system.  In the next tutorial, we'll get more comfortable with playing around with kernel state.]]> Holding a critical section during SendMessage https://docs.microsoft.com/archive/blogs/meason/holding-a-critical-section-during-sendmessage Wed, 23 Jun 2010 11:31:00 GMT https://blogs.msdn.microsoft.com/meason/2010/06/23/holding-a-critical-section-during-sendmessage/ Can you spot the defect with the following code?

EnterCriticalSection(&cs);
SendMessage(hwnd, WM_MYMESSAGE, 0, 0);
LeaveCriticalSection(&cs);

The problem is that the SendMessage() is a blocking call, and Windows can do many things from within this call.  This creates a situation that is highly prone to deadlocking.

In the best case scenario, the application will remain responsive but its performance will suffer.  The best case is when the target HWND is owned by the calling thread and Windows does not perform any additional message dispatching.  In that case, the WNDPROC is invoked and it is up to the handler to determine how long the call will take.  All other threads that attempt to acquire the critical section become blocked and application performance suffers.

 In the worst case scenario, a deadlock will happen.  If the target HWND is on another thread, the call will block until the other thread pumps its message queue and handles the message.  However, the other thread may be blocked from pumping its message queue if it is trying to acquire the same critical section.

That simple code above may be an obvious deadlock situation.  The situation that is less obvious and more frequently leads to hangs is when you are calling into third party code and are unsure about its implementation details.  For example, the following code may be prone to deadlocking without being so obvious:

EnterCriticalSection(&cs);
pThirdPartyObject->InvokeFunction();
LeaveCriticalSection(&cs);

 The third party code may call something like SendMessage in its implementation, which could lead to a deadlock.

The lesson to learn is: Be careful about what you do while you are holding a synchronization primitive such as a critical section.  Minimize the amount of code between the acquire and release.

]]>