Dump Diving into Cert Failure: How to Get a Good Callstack from TriageDump.dmp

  • Article

As of this writing, I am currently working on a support case where the developer's app has been failing certification for a FileNotFoundException in his application that has been submitted to the Windows Store.  He has been trying to understand exactly where in his source code the application is failing.  To do this, he's been using Visual Studio to open up the TriageDump.dmp file which was sent as part of the certification failure report.  The developer reported that he was not able to get a good callstack from the dump file, so I took a crack at it.  The following describes the process by which I was able to get to the callstack where the exception occurred.

I want to note that I used WinDbg, a free downloadable tool from Microsoft.  It is a standalone, text-driven debugging engine which is widely used by Microsoft employees for most debugging scenarios.  The same debugging engine is included with Visual Studio, and can be used if you go to Open > Dump File.

*Note: name of the app has been changed for privacy purposes.

In case you're not familiar with it, the Windows Store Certification report contains a Report.cab file.  This is simply a zip archive file, and files can be extracted from the .cab exact like a zip.  The cab file contains a file called triagedump.dmp, which is a snapshot of the app at the exact time that the app crashed due to the exception that occurred.

In the interest of keeping this blog post manageable, I will show the process and commands I used to get to the usable callstack, and explain what they do, rather than explain the intricacies of why. You can do this using your own dumps and apps.

  1. Open the triagedump.dmp file  in WinDbg.  The command line at the bottom of the page: 

    0:019>

    indicates the current thread context.  WinDbg typically opens to the thread where the exception occurred.

    FYI: The first zero refers to the process, which can change if you were debugging the kernel.  It is always 0 for a crash dump.  The 019 refer to the number of the current thread context inside the process.
     

  2. Set the symbols path:     

    0:019> !symfix

    Symbols are files that help the debugger translate the binary information inside the dump to human-readable information. This particular command sets a default symbols path.  For WinDbg, it downloads from the public Microsoft symbols server: : https://msdl.microsoft.com/download/symbols

  3. Make it so you can tell where the symbols are being loaded from. 

    0:019> !sym noisy

    This is important so we can tell where to place our app-specific symbols and images so that WinDbg can find them.

  4. We have tell WinDbg to reload the symbols and images since we have a new symbols path: 

    0:019>.reload

  5. WinDbg will perform an automated analysis on the dump:

    0:019> !analyze –v

    This will take a long time and you will see a bunch of debugging spew.  Eventually you will get some interesting information, including a callstack and an exception object stack, but it won't be helpful yet.  In this case, the exception object stack actually had the function call that caused the exception:

    EXCEPTION_OBJECT: !pe 25afc54
    Exception object: 025afc54
    Exception type:  System.IO.FileNotFoundException
    Message: <Invalid Object>
    InnerException: <none>
    StackTrace (generated):   

    SP IP       Function
    04F0F364 6F00C946 mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)+0x5e
    04F0F374 6F00C8D9 mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x35
    04F0F380 6EBB8498 APPNAME_ni!UNKNOWN+0x270
    04F0F4BC 6F79E6FF mscorlib_ni!System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__0(System.Object)+0x33
    04F0F4C4 6D2E13D4 System_Runtime_WindowsRuntime_ni!System.Threading.WinRTSynchronizationContext+Invoker.InvokeCore()+0x24
    0A25F4B4 6F6A2607 mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x17
    0A25F4BC 6D30E728 System_Runtime_WindowsRuntime_ni!System.Threading.WinRTSynchronizationContext+Invoker.<InvokeCore>b__0(System.Object)+0x28
    0A25F4C0 6EFACB6A mscorlib_ni!System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)+0x3e
    0A25F4C8 6EFC2407 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xa7
    0A25F534 6EFC2346 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,System.Threading.ContextCallback, System.Object, Boolean)+0x16
    0A25F548 6EFCDE60 mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+0x60
    0A25F55C 6EFCD609 mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+0x149
    0A25F5AC 6EFCD4A5 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()+0x5

    The highlighted line is where the app in question threw the exception. The problem is that there's no information that says the specific method inside the app where the problem occurred.  That's why we needed the "!sym noisy" command.  We need to figure out what more information is needed for the debugger to give us more information. Let's find this information in the rest of the spew:

    C:\ProgramData\dbg\sym\APPNAME.exe\51D597BF64000\APPNAME.exe not found

    What you want to do is create the path to this location, then put the exact executable that matches to this dump inside that path.  You can ensure that the correct .exe matches up by checking the green number in the path with the timestamp of the .exe.  You can check the timestamp of the .exe that you are using by doing this:

    Open a new WinDbg instance. 

    1. File > Open Crash Dump…
      Change “Crash Dump Files” to “All Files”
      Navigate to the “.exe” file
      > OK

    2. At the command line:

      0x000> lmv  

      You will see this:

      start end module name
      00400000 00464000 APPNAMEC (no symbols)          
      Loaded symbol image file: APPNAME.exe
      Mapped memory image file: C:\ProgramData\dbg\sym\APPNAME.exe\51D597BF64000\APPNAME.exe
      Image path: C:\ProgramData\dbg\sym\APPNAME.exe\51D597BF64000\APPNAME.exe
      Image name:APPNAME.exe

      Using CLR debugging support for all symbols
      Has CLR image header, track-debug-data flag not set

      Timestamp: Thu Jul 04 11:41:51 2013 (51D597BF)
      CheckSum: 00000000
      ImageSize: 00064000
      File version: 1.0.0.0
      Product version:  1.0.0.0
      File flags: 0 (Mask 3F)
      File OS: 4 Unknown Win32
      File type: 1.0 App
      File date: 00000000.00000000   
      Translations: 0000.04b0
      CompanyName: Windows Store App Developer
      ProductName: APPNAME
      InternalName: APPNAME.exe
      OriginalFilename: APPNAME.exe
      ProductVersion:   1.0.0.0
      FileVersion: 1.0.0.0
      FileDescription: APPNAME
      LegalCopyright:   Copyright © 2013
      Comments: App on Windows 8

      The green numbers put together should match up to the location where the .exe is being looked for.

  6. You can get the exact image (.exe) from the .appx package.  Rename .appx to .zip and unzip the content.  You will see the .exe inside that package under the .appx folder.

  7. After you’ve put the exact image in that folder location, run

    0x019> .reload

    To pull in the new image.

  8. Run this to get a refreshed callstack
      
    0x019> !analyze –v

  9. Now you should get much better debugging information and a good callstack:

    BUGCHECK_STR: APPLICATION_FAULT_CLR_EXCEPTION__SYSTEM.IO.FILENOTFOUNDEXCEPTION__SYSTEM.IO.FILENOTFOUNDEXCEPTION__SYSTEM.IO.FILENOTFOUNDEXCEPTION
    PRIMARY_PROBLEM_CLASS: CLR_EXCEPTION_SYSTEM.IO.FILENOTFOUNDEXCEPTION
    DEFAULT_BUCKET_ID: CLR_EXCEPTION_SYSTEM.IO.FILENOTFOUNDEXCEPTION

    STACK_TEXT:
    04f0f364 6f00c946 mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess+0x5e
    04f0f374 6f00c8d9 mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification+0x35
    04f0f380 6ebb8498 APPNAME_ni!APPNAME.Controls.Layouts.PageLayouts+_LoadLiveOverlays_d__0.MoveNext+0x270
    04f0f4bc 6f79e6ff mscorlib_ni!System.Runtime.CompilerServices.AsyncMethodBuilderCore._ThrowAsync_b__0+0x33
    04f0f4c4 6d2e13d4 system_runtime_windowsruntime_ni!System.Threading.WinRTSynchronizationContext+Invoker.InvokeCore+0x24
    0a25f4b4 6f6a2607 mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw+0x17
    0a25f4bc 6d30e728 system_runtime_windowsruntime_ni!System.Threading.WinRTSynchronizationContext+Invoker._InvokeCore_b__0+0x28
    0a25f4c0 6efacb6a mscorlib_ni!System.Threading.QueueUserWorkItemCallback.WaitCallback_Context+0x3e
    0a25f4c8 6efc2407 mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xa7
    0a25f534 6efc2346 mscorlib_ni!System.Threading.ExecutionContext.Run+0x16
    0a25f548 6efcde60 mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem+0x60
    0a25f55c 6efcd609 mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch+0x149
    0a25f5ac 6efcd4a5 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback+0x5

    Notice that the highlighted line gives far more information that it did previously. You can now look at the specific method in the application to determine why the application is crashing.

I hope that this process helps you understand how you can get useful information from the dumps that are provided by the certification team. Please leave questions in the comments below.

Special thanks to Jeff Sanders for providing lots of valuable input on the debugging process!

Be sure to follow me on Twitter: @WinDevMatt