June 2011

Volume 26 Number 06

Debugger Engine API - Writing a Debugging Tools for Windows Extension, Part 3: Clients and Callbacks

By Andrew Richards | June 2011

In this third installment about the Debugger API, I’m going to delve deeper into the relationship a debugger extension can have with the debugger. I’ll give you an overview of the architecture of debugger clients and debugger callbacks. In doing so, we’ll get into the nitty-gritty of the DEBUG_OUTPUT_XXX and DEBUG_OUTCTL_XXX constants.

I’ll use this foundation to implement an encapsulation of the Son of Strike (SOS) debugger extension. I’ll enhance the SOS output with Debugger Markup Language (DML) and demonstrate how you can leverage the built-in debugger commands and other extensions to retrieve information required by your extensions.

Before continuing, be sure you’ve read the previous two installments to understand what a debugger extension is (including how I’m building and testing the examples in this article) and how to correctly produce output. They can be found at msdn.microsoft.com/magazine/gg650659 and msdn.microsoft.com/magazine/hh148143.

Output Control

There are two ways to produce output in an extension: the IDebugControl::Output* functions and the IDebugControl::ControlledOutput* functions. The Output functions are just a simplification of the ControlledOuput functions. They retrieve the current output control setting and pass this to ControlledOutput as the first parameter:

HRESULT ControlledOutput(
  [in]  ULONG OutputControl,
  [in]  ULONG Mask,
  [in]  PCSTR Format,
         ...
);

I gave an overview of the DEBUG_OUTCTL_XXX constants used for the OutputControl parameter in the second article. In this article, I’m going to focus on the lower bits that control the scope of the output. One of the DEBUG_OUTCTL_XXX scope-based constants must be used for the lower bits. The value directs where the output is to go. This can be to all debugger clients (DEBUG_OUTCTL_ALL_CLIENTS), just the IDebugClient associated with the IDebugControl interface (DEBUG_OUTCTL_THIS_CLIENT), all other clients (DEBUG_OUTCTL_ALL_OTHER_CLIENTS), nowhere at all (DEBUG_OUTCTL_IGNORE) or just the log file (DEBUG_OUTCTL_LOG_ONLY).

In Example06 in the accompanying code download (which is the same as the previous article), the !outctlpassed command (see Figure 1) is based on the passed IDebugClient interface.

Figure 1 !outctlpassed

HRESULT CALLBACK 
outctlpassed(PDEBUG_CLIENT pDebugClient, PCSTR args)
{
  UNREFERENCED_PARAMETER(args);

  IDebugControl* pDebugControl;
  if (SUCCEEDED(pDebugClient->
    QueryInterface(__uuidof(IDebugControl), (void **)&pDebugControl)))
  {
    pDebugControl->ControlledOutput(DEBUG_OUTCTL_THIS_CLIENT,           
      DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_THIS_CLIENT\n");
    pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_CLIENTS,       
      DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_ALL_CLIENTS\n");
    pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_OTHER_CLIENTS,  
      DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_ALL_OTHER_CLIENTS\n");
    pDebugControl->ControlledOutput(DEBUG_OUTCTL_IGNORE,             
      DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_IGNORE\n");
    pDebugControl->ControlledOutput(DEBUG_OUTCTL_LOG_ONLY,          
      DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_LOG_ONLY\n");

    pDebugControl->Release();
  }
  return S_OK;
}

The function does a QueryInterface for the IDebugControl interface (of the passed IDebugClient). Then ControlledOutput is called with each of the output control scopes. The Mask is set to normal (DEBUG_OUTPUT_NORMAL) so that it isn’t excluded from output. By default, only a Mask value of verbose (DEBUG_OUTPUT_VERBOSE) is excluded from the WinDBG Output window. To test the command, I use the test_windbg.cmd script. In the launched WinDBG, I open the log file (.logopen), run the !outctlpassed command and close the log file (.logclose), as shown here:

0:000> .logopen outctlpassed.txt
Opened log file 'outctlpassed.txt'
0:000> !outctlpassed
DEBUG_OUTCTL_THIS_CLIENT
DEBUG_OUTCTL_ALL_CLIENTS
0:000> .logclose
Closing open log file outctlpassed.txt

The !outctlpassed command only includes DEBUG_OUTCTL_THIS_CLIENT and DEBUG_OUTCTL_ALL_CLIENTS output. The DEBUG_OUTCTL_ALL_OTHER_CLIENTS, DEBUG_OUTCTL_IGNORE and DEBUG_OUTCTL_LOG_ONLY output controls are omitted in the Output window. But if you look in the log file, there’s a different result, as shown here:

Opened log file 'outctlpassed.txt'
0:000> !outctlpassed
DEBUG_OUTCTL_THIS_CLIENT
DEBUG_OUTCTL_ALL_CLIENTS
DEBUG_OUTCTL_ALL_OTHER_CLIENTS
DEBUG_OUTCTL_LOG_ONLY
0:000> .logclose
Closing open log file outctlpassed.txt

The only output control missing from the log file is DEBUG_OUTCTL_IGNORE.

Debugger Architecture

To understand what happened to the missing output in the WinDBG Output window and the log file, we need to delve in to the debugger architecture. Debugging is based on four layers: the Debugger Helper, the Debugger Engine, the Debugger Client and Debugger Extensions.

At the bottom is the Debugger Helper (dbghelp.dll). This library contains all of the functionality to resolve symbols.

The next layer is the Debugger Engine (dbgeng.dll). This library handles the debugging session and, in particular, provides the support to debug a remote target. Included in this library is the handling of the debugger interfaces (for example, IDebugClient, IDebugControl and so on).

The next layer is the Debugger Client (client) layer. At this layer, input and output are processed as the client sees fit. The WinDBG and NTSD debuggers are in this layer. A client uses the Debugger Engine DebugCreate and DebugConnect functions to create an IDebugClient object that’s attached to the desired target. The target can be local or remote (through the Debugger Engine proxy support).

The last layer is Debugger Extensions (extension). A client invokes commands exported by the Debugger Extension DLL. The IDebugClient interface that the Debugger Client receives from the Debugger Engine is the same interface that the client passes to extensions. Although invoked by a client, an extension only interacts with the Debugger Engine. Extensions can share the IDebugClient interface with the client as long as they don’t damage its configuration. Alternatively, they can make their own client (via IDebugClient::CreateClient) and configure it as needed.

Multiple Debugger Clients

Multiple clients (IDebugClient objects) can be attached to the same debugging session; it’s the session that’s singular, not the clients. Each client has its own private interface to the Debugger Engine, and it’s this ability that we’re going to leverage.

Initially, when you attach WinDBG or NTSD to a process, there’s just one client—the IDebugClient interface returned by DebugCreate or DebugConnect. When an extension calls IDebugClient::CreateClient against this returned interface, the Debugger Engine makes another client that’s associated with the session. The new client is attached to the same debugging session, but it’s in a default state. Callbacks aren’t set on it and the output mask is the default value (everything except Verbose).

In Figure 2, the !outctlcreate does the same output as !outctlpassed, but is based on a newly created IDebugClient object.

Figure 2 !outctlcreate

HRESULT CALLBACK 
outctlcreate(PDEBUG_CLIENT pDebugClient, PCSTR args)
{
  UNREFERENCED_PARAMETER(args);

  IDebugClient* pDebugCreated;
  if (SUCCEEDED(pDebugClient->CreateClient(&pDebugCreated)))
  {
    IDebugControl* pDebugControl;
    if (SUCCEEDED(pDebugCreated->QueryInterface(__uuidof(IDebugControl), 
      (void **)&pDebugControl)))
    {
      pDebugControl->ControlledOutput(DEBUG_OUTCTL_THIS_CLIENT,        
        DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_THIS_CLIENT\n");
      pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_CLIENTS,       
        DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_ALL_CLIENTS\n");
      pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_OTHER_CLIENTS, 
        DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_ALL_OTHER_CLIENTS\n");
      pDebugControl->ControlledOutput(DEBUG_OUTCTL_IGNORE,            
        DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_IGNORE\n");
      pDebugControl->ControlledOutput(DEBUG_OUTCTL_LOG_ONLY,          
        DEBUG_OUTPUT_NORMAL, "DEBUG_OUTCTL_LOG_ONLY\n");
      pDebugControl->Release();
    }
    pDebugCreated->Release();
  }
  return S_OK;
}

As before, to test the command, I use the test_windbg.cmd script. In the launched WinDBG, I open the log file (.logopen), run the !outctlcreate command and close the log file (.logclose), as shown here:

0:000> .logopen outctlcreate.txt
Opened log file 'outctlcreate.txt'
0:000> !outctlcreate
DEBUG_OUTCTL_ALL_CLIENTS
DEBUG_OUTCTL_ALL_OTHER_CLIENTS
0:000> .logclose
Closing open log file outctlcreate.txt

The !outctlcreate command only includes the DEBUG_OUTCTL_ALL_CLIENTS and DEBUG_OUTCTL_ALL_OTHER_CLIENTS output. The “all others” output replaces the “this” output because the WinDBG client is now the “other” client. The log file results are still the same, as shown here:

Opened log file 'outctlcreate.txt'
0:000> !outctlcreate
DEBUG_OUTCTL_THIS_CLIENT
DEBUG_OUTCTL_ALL_CLIENTS
DEBUG_OUTCTL_ALL_OTHER_CLIENTS
DEBUG_OUTCTL_LOG_ONLY
0:000> .logclose
Closing open log file outctlcreate.txt

Inside the Debugging Engine, the ControlledOutput call is being conditionally acted upon. The engine invokes the output callback on all clients that match the output control criteria. So, if you make your own client and use the DEBUG_OUTCTL_THIS_CLIENT output control, the output will stay local (that is, not displayed in WinDBG). Note that the output still goes to the log file. If you add the “not logged” high bit (DEBUG_OUTCTL_NOT_LOGGED), you can stop this, too.

Once again, the !outctlthis command does just “this” and “not logged” output (DEBUG_OUTCTL_THIS_CLIENT | DEBUG_OUTCTL_NOT_LOGGED) on a created client, as shown in Figure 3.

Figure 3 !outctlthis

HRESULT CALLBACK 
outctlthis(PDEBUG_CLIENT pDebugClient, PCSTR args)
{
  UNREFERENCED_PARAMETER(args);

  IDebugClient* pDebugCreated;
  if (SUCCEEDED(pDebugClient->CreateClient(&pDebugCreated)))
  {
    IDebugControl* pDebugControl;
    if (SUCCEEDED(pDebugCreated->QueryInterface(__uuidof(IDebugControl), 
      (void **)&pDebugControl)))
    {
      pDebugControl->ControlledOutput(DEBUG_OUTCTL_THIS_CLIENT |  
        DEBUG_OUTCTL_NOT_LOGGED, DEBUG_OUTPUT_NORMAL, 
        "DEBUG_OUTCTL_THIS_CLIENT | DEBUG_OUTCTL_NOT_LOGGED\n");
      pDebugControl->Release();
    }
    pDebugCreated->Release();
  }
  return S_OK;
}

As before, to test the command, I use the test_windbg.cmd script. In the launched WinDBG, I open the log file (.logopen), run the !outctlthis command and close the log file (.logclose), as shown here:

0:000> .logopen outctlthis.txt
Opened log file 'outctlthis.txt'
0:000> !outctlthis
0:000> .logclose
Closing open log file outctlthis.txt

The log file doesn’t record the command’s output this time. Note that it does record the execution because prompt output is being created by the WinDBG client prior to the invoking of the extension, as shown here:

Opened log file 'outctlthis.txt'
0:000> !outctlthis
0:000> .logclose
Closing open log file outctlthis.txt

In effect, I’ve redirected output to NULL because the client created doesn’t have any associated output callback—not exactly a useful thing. But if I change the code to use the IDebugControl::Execute function, I can now execute any command without it being displayed in WinDBG or being logged, as shown here:

pDebugControl->Execute(DEBUG_OUTCTL_THIS_CLIENT | DEBUG_OUTCTL_NOT_LOGGED, 
  args, DEBUG_EXECUTE_NOT_LOGGED | DEBUG_EXECUTE_NO_REPEAT);

When using Execute in this way, you need to set the DEBUG_EXECUTE_NOT_LOGGED and DEBUG_EXECUTE_NO_REPEAT flags so that the prompt output isn’t generated in the log file and the command isn’t saved for repetition (for example, for when the user executes a blank command by pressing Enter in the command prompt).

As shown in Figure 4, at the end the !outctlexecute command silently executes the supplied argument.

Figure 4 !outctlexecute

HRESULT CALLBACK 
outctlexecute(PDEBUG_CLIENT pDebugClient, PCSTR args)
{
  IDebugClient* pDebugCreated;
  if (SUCCEEDED(pDebugClient->CreateClient(&pDebugCreated)))
  {
    IDebugControl* pDebugControl;
    if (SUCCEEDED(pDebugCreated->QueryInterface(__uuidof(IDebugControl), 
      (void **)&pDebugControl)))
  {
    pDebugControl->Execute(DEBUG_OUTCTL_THIS_CLIENT | DEBUG_OUTCTL_NOT_LOGGED, 
      args, DEBUG_EXECUTE_NOT_LOGGED | DEBUG_EXECUTE_NO_REPEAT);
    pDebugControl->Release();
  }
  pDebugCreated->Release();
  }
  return S_OK;
}

I’m executing a forced reload of symbols (.reload /f). This would usually produce lots of symbol output, but !outctlexecute has hidden the output, as shown here:

0:000> !outctlexecute .reload /f

Output Callbacks

When the Debugger Engine routes output requests to the clients (indicated by the output control), there are two additional constraints applied:

  • Is the output of a type (mask) that’s wanted?
  • Is there an associated output callback?

When IDebugControl::Output or IDebugControl::ControlledOutput is invoked, the Debugging Engine compares the supplied mask with the client’s output mask (IDebugClient::GetOutputMask). The two masks need to match for the output callback to be invoked.

Next, the Debugging Engine calls IDebugClient::GetOutputCallbacks to get an output callback interface (IDebugOutputCallbacks, IDebugOutputCallbacks2 or IDebugOutputCallbacksWide). If there’s an output callback set on the client (previously done via IDebugClient::SetOutputCallbacks), the Debugging Engine invokes the callback—for example, IDebugOutputCallbacks::Output.

In Example07 in the accompanying code download, I’ve coded an IDebugOutputCallbacks implementation that supports 1MB of normal output and 1MB of error output. (The header is in Figure 5 and the code is in Figure 6.)

Figure 5 OutputCallbacks.h

#ifndef __OUTPUTCALLBACKS_H__
#define __OUTPUTCALLBACKS_H__

#include "dbgexts.h"

class COutputCallbacks : public IDebugOutputCallbacks
{
  private:
    long m_ref;
    PCHAR m_pBufferNormal;
    size_t m_nBufferNormal;
    PCHAR m_pBufferError;
    size_t m_nBufferError;

  public:
    COutputCallbacks()
     {
      m_ref = 1;
      m_pBufferNormal = NULL;
      m_nBufferNormal = 0;
      m_pBufferError = NULL;
      m_nBufferError = 0;
    }

    ~COutputCallbacks()
    {
      Clear();
    }

    // IUnknown
    STDMETHOD(QueryInterface)(__in REFIID InterfaceId, __out PVOID* Interface);
    STDMETHOD_(ULONG, AddRef)();
    STDMETHOD_(ULONG, Release)();

    // IDebugOutputCallbacks
    STDMETHOD(Output)(__in ULONG Mask, __in PCSTR Text);

    // Helpers
    ULONG SupportedMask() { return DEBUG_OUTPUT_NORMAL | DEBUG_OUTPUT_ERROR; }
    PCHAR BufferNormal() { return m_pBufferNormal; }
    PCHAR BufferError() { return m_pBufferError; }
    void Clear();   
};

#endif // #ifndef __OUTPUTCALLBACKS_H__

Figure 6 OutputCallbacks.cpp

#include "dbgexts.h"
#include "outputcallbacks.h"

#define MAX_OUTPUTCALLBACKS_BUFFER 0x1000000  // 1Mb
#define MAX_OUTPUTCALLBACKS_LENGTH 0x0FFFFFF  // 1Mb - 1

STDMETHODIMP COutputCallbacks::QueryInterface(__in REFIID InterfaceId, 
  __out PVOID* Interface)
{
  *Interface = NULL;
  if (IsEqualIID(InterfaceId, __uuidof(IUnknown)) || IsEqualIID(InterfaceId, 
    __uuidof(IDebugOutputCallbacks)))
  {
    *Interface = (IDebugOutputCallbacks *)this;
    InterlockedIncrement(&m_ref);
    return S_OK;
  }
  else
  {
    return E_NOINTERFACE;
  }
}

STDMETHODIMP_(ULONG) COutputCallbacks::AddRef()
{
  return InterlockedIncrement(&m_ref);
}

STDMETHODIMP_(ULONG) COutputCallbacks::Release()
{
  if (InterlockedDecrement(&m_ref) == 0)
  {
    delete this;
    return 0;
  }
  return m_ref;
}

STDMETHODIMP COutputCallbacks::Output(__in ULONG Mask, __in PCSTR Text)
{
  if ((Mask & DEBUG_OUTPUT_NORMAL) == DEBUG_OUTPUT_NORMAL)
  {
    if (m_pBufferNormal == NULL)
    {
      m_nBufferNormal = 0;
      m_pBufferNormal = (PCHAR)malloc(sizeof(CHAR)*(MAX_OUTPUTCALLBACKS_BUFFER));
      if (m_pBufferNormal == NULL) return E_OUTOFMEMORY;
      m_pBufferNormal[0] = '\0';
      m_pBufferNormal[MAX_OUTPUTCALLBACKS_LENGTH] = '\0';
    }
    size_t len = strlen(Text);
    if (len > (MAX_OUTPUTCALLBACKS_LENGTH-m_nBufferNormal))
    {
      len = MAX_OUTPUTCALLBACKS_LENGTH-m_nBufferNormal;
    }
    if (len > 0)
    {
      memcpy(&m_pBufferNormal[m_nBufferNormal], Text, len);
      m_nBufferNormal += len;
      m_pBufferNormal[m_nBufferNormal] = '\0';
    }
  }
  if ((Mask & DEBUG_OUTPUT_ERROR) == DEBUG_OUTPUT_ERROR)
  {
    if (m_pBufferError == NULL)
    {
      m_nBufferError = 0;
      m_pBufferError = (PCHAR)malloc(sizeof(CHAR)*(MAX_OUTPUTCALLBACKS_BUFFER));
      if (m_pBufferError == NULL) return E_OUTOFMEMORY;
      m_pBufferError[0] = '\0';
      m_pBufferError[MAX_OUTPUTCALLBACKS_LENGTH] = '\0';
    }
    size_t len = strlen(Text);
    if (len >= (MAX_OUTPUTCALLBACKS_LENGTH-m_nBufferError))
    {
      len = MAX_OUTPUTCALLBACKS_LENGTH-m_nBufferError;
    }
    if (len > 0)
    {
      memcpy(&m_pBufferError[m_nBufferError], Text, len);
      m_nBufferError += len;
      m_pBufferError[m_nBufferError] = '\0';
    }
  }
  return S_OK;
}

void COutputCallbacks::Clear()
{
  if (m_pBufferNormal)
  {
    free(m_pBufferNormal);
    m_pBufferNormal = NULL;
    m_nBufferNormal = 0;
  }
  if (m_pBufferError)
  {
    free(m_pBufferError);
    m_pBufferError = NULL;
    m_nBufferError = 0;
  }
}

The class appends the string to the normal or error buffer upon each Output call. Apart from the functions required for the IUnknown and IDebugOutputCallbacks interfaces, there are some additional functions to provide access to the buffers (BufferNormal and BufferError), to clear (Clear) them, and a function to get the supported output types (SupportedMask).

Each time Output or ControlledOutput is invoked (and the IDebugClient criteria are met), the callback’s Output function is called. The mask passed is the same mask that’s set on the original output call. My Output function does bitmask comparison on the passed mask, so that the text is associated with the correct buffer. It’s up to the implementation of the callback to decide whether the output is stored separately, combined or ignored. A single IDebugClient::Execute call might result in hundreds of Output calls of varying types. Because I want to interpret the normal and error output of the entire Execute operation, I concatenate these strings into two associated buffers.

The IDebugOutputCallbacks interface is passed the string in ANSI and TEXT format. If the original text is in Unicode or in DML, the engine will do a conversion prior to the callback. To capture Unicode (wide) content, use the IDebugOutCallbacksWide interface instead. The only difference between these two interfaces is the Text parameter being passed as PCWSTR instead of PCSTR. Both of these interfaces only support the notification of TEXT-based output. They don’t receive DML-formatted text.

To get DML content, you need to use the IDebugOutputCallbacks2 interface. This interface gives you control of how the text received by the callback is formatted: TEXT, DML or either one. The GetInterestMask function is used to define this. It’s called when the interface is set on a client via IDebugClient::SetOutputCallbacks. You return whether you want to receive TEXT (DEBUG_OUTCBI_TEXT), DML (DEBUG_OUTCBI_DML) or either type of output (DEBUG_OUTCBI_ANY_FORMAT). Note that this isn’t the same as the Output Mask.

The Output function has no purpose on this interface; it’s an artifact of the (fake) interface inheritance of IDebugCallbacks. All output callback notification is via the Output2 function:

STDMETHOD(Output2)(ULONG Which, ULONG Flags, ULONG64 Arg, PCWSTR Text)

The Which parameter specifies whether TEXT (DEBUG_OUTCB_TEXT) or DML (DEBUG_OUTCB_DML) is being passed, or that a flush (DEBUG_OUTCB_EXPLICIT_FLUSH) is desired (IDebugClient::FlushCallbacks). Note that these are the CB* constants; not the CBI* constants used for the interest mask.

The Flags parameter specifies additional information in a bitmask. In particular, for DML content it specifies whether the content contains hyperlinks (DEBUG_OUTCBF_DML_HAS_TAGS) or has special characters (", &, < and >) that are encoded (DEBUG_OUTCBF_DML_HAS_SPECIAL_CHARACTERS). The parameter can also indicate that a flush (DEBUG_OUTCBF_COMBINED_EXPLICIT_FLUSH) is desired after the output has been processed. This occurs when Output and FlushCallbacks calls are combined by the engine.

The Arg parameter contains the output mask for DML, TEXT and FLUSH callbacks.

Finally, the Text parameter is the text being outputted. This will be NULL when the callback is being invoked for a flush (DEBUG_OUTCB_EXPLICIT_FLUSH). The text is always passed as Unicode.

Execution

It doesn’t take much code to add the callback to the !outctlexecute command (see Figure 4). The additional steps are to allocate the callback and to associate it with the created IDebugClient interface before the call to Execute.

I implement my callbacks as referenced counted heap objects. If you’re using the Windows Driver Kit build environment, you’ll need to disable “PREfast for Drivers” warning 28197 so that you don’t get an OACR (Microsoft Auto Code Review) warning about the “new” allocation each time you compile the project. (PREfast doesn’t detect that the class is reference counted.)

There are two parts to the association of the callback to the client: the output mask (SetOutputMask) and the output callback (SetOutputCallbacks). I have a helper function on my callback class to return the supported output mask (COutputCallbacks::SupportedMask) so that I don’t have to hardcode it. If I didn’t set the mask, I’d run the risk of not receiving the expected output.

The !callbackecho command (see Figure 7) is implemented in the Example07 project in the accompanying download.

Figure 7 !callbackecho

HRESULT CALLBACK 
callbackecho(PDEBUG_CLIENT pDebugClient, PCSTR args)
{
  IDebugClient* pDebugCreated;
  if (SUCCEEDED(pDebugClient->CreateClient(&pDebugCreated)))
  {
    IDebugControl* pDebugControl;
    if (SUCCEEDED(pDebugCreated->QueryInterface(__uuidof(IDebugControl), 
      (void **)&pDebugControl)))
    {
      // Create our IDebugOutputCallbacks-based object
      #pragma warning( disable : 28197 ) // PreFAST sees the 'new' as a leak due to 
                                     // the use of AddRef/Release
      COutputCallbacks* pOutputCallbacks = new COutputCallbacks;
      if (pOutputCallbacks != NULL)
      {
        // Apply IDebugOutputCallbacks to the IDebugClient
        if (SUCCEEDED(pDebugCreated->
          SetOutputMask(pOutputCallbacks->SupportedMask())) &&   
          SUCCEEDED(pDebugCreated->SetOutputCallbacks(pOutputCallbacks)))
      {
        // Execute and display 'vertarget'
        pDebugControl->Execute(DEBUG_OUTCTL_THIS_CLIENT | DEBUG_OUTCTL_NOT_LOGGED, 
          "vertarget", DEBUG_EXECUTE_NOT_LOGGED | DEBUG_EXECUTE_NO_REPEAT);
        pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_OTHER_CLIENTS, 
          DEBUG_OUTPUT_NORMAL, "[Normal]\n%s\n", pOutputCallbacks->BufferNormal() ? 
            pOutputCallbacks->BufferNormal() : "");
        pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_OTHER_CLIENTS, 
          DEBUG_OUTPUT_ERROR , "[Error] \n%s\n", pOutputCallbacks->BufferError() ? 
          pOutputCallbacks->BufferError() : "");

        // Clear the callback buffers
        pOutputCallbacks->Clear();

        // Execute and display the passed command
        pDebugControl->Execute(DEBUG_OUTCTL_THIS_CLIENT | DEBUG_OUTCTL_NOT_LOGGED, 
          args, DEBUG_EXECUTE_NOT_LOGGED | DEBUG_EXECUTE_NO_REPEAT);
        pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_OTHER_CLIENTS, 
          DEBUG_OUTPUT_NORMAL, "[Normal]\n%s\n", pOutputCallbacks->BufferNormal() ? 
          pOutputCallbacks->BufferNormal() : "");
        pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_OTHER_CLIENTS, 
          DEBUG_OUTPUT_ERROR , "[Error] \n%s\n", pOutputCallbacks->BufferError() ? 
          pOutputCallbacks->BufferError() : "");

        pDebugCreated->SetOutputCallbacks(NULL); // It's a best practice to 
                                                // release callbacks 
                                                // before the client release
        }
        pOutputCallbacks->Release();
      }
      pDebugControl->Release();
    }
    pDebugCreated->Release();
  }
  return S_OK;
}

It executes two commands (“vertarget” and the passed argument) and displays the normal and error output of each. When using the ControlledOutput function, you’ll note that I use the DEBUG_OUTCTL_ALL_OTHER_CLIENTS output control. This directs the output to all other clients and avoids the output being captured by my associated output callback. If you’re looping though captured output and are producing output of the same mask, be sure that you use the DEBUG_OUTCTL_ALL_OTHER_CLIENTS output control, otherwise you could easily get yourself into an infinite loop.

As before, I use the test_windbg.cmd script to test the command. In the launched WinDBG, I run the !callbackecho command and see the normal output of “vertarget” and the error output of “r @rip.” (The register command fails because this is an x86 target and “rip” is a x64 register.) Note that in between the Execute calls, I call my Clear helper function to reset the buffers. This is what causes the “vertarget” output to be missing from the “r @rip” command output, as shown here:

0:000> !callbackecho r @rip
[Normal]
Windows 7 Version 7600 MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
kernel32.dll version: 6.1.7600.16385 (win7_rtm.090713-1255)
Machine Name:
Debug session time: Sat Jan 29 22:01:59.080 2011 (UTC - 8:00)
System Uptime: 0 days 6:22:48.483
Process Uptime: 0 days 0:01:05.792
Kernel time: 0 days 0:00:00.015
User time: 0 days 0:00:00.000
 
[Error]
 
[Normal]
 
[Error]
           ^ Bad register error in 'r @rip'

You can actually implement the same functionality without making a client of your own. If you carefully swap in the output mask and the output callback of the passed client with your own values, you can achieve the same result. However, you need to be quite accurate when doing this. The passed IDebugClient is the client that the (WinDBG) debugger uses for output to the Output window. If you don’t return it to the state that it started at, the debugger will no longer function correctly.

Figure 8 (from Example07 in the code download) shows the toggling approach using the !commandtoggle command.

Figure 8 !callbacktoggle

HRESULT CALLBACK 
callbacktoggle(PDEBUG_CLIENT pDebugClient, PCSTR args)
{
  IDebugControl* pDebugControl;
  if (SUCCEEDED(pDebugClient->QueryInterface(__uuidof(IDebugControl), 
    (void **)&pDebugControl)))
  {
    // Remember the original mask and callback
    ULONG ulOriginalMask;
    IDebugOutputCallbacks* pOriginalCallback;
    if (SUCCEEDED(pDebugClient->GetOutputMask(&ulOriginalMask)) &&
      SUCCEEDED(pDebugClient->GetOutputCallbacks(&pOriginalCallback)))
    {
      // Create our IDebugOutputCallbacks based object
      #pragma warning( disable : 28197 ) // PreFAST sees the 'new' as a leak due to 
                                     // the use of AddRef/Release
      COutputCallbacks* pOutputCallbacks = new COutputCallbacks;
      if (pOutputCallbacks != NULL)
      {
        // Apply IDebugOutputCallbacks to the IDebugClient
        if (SUCCEEDED(pDebugClient->SetOutputMask(pOutputCallbacks->
          SupportedMask())) && (pDebugClient->
          SetOutputCallbacks(pOutputCallbacks)))
        {
          // Execute 'vertarget'
          pDebugControl->Execute(DEBUG_OUTCTL_THIS_CLIENT | 
            DEBUG_OUTCTL_NOT_LOGGED, "vertarget", 
            DEBUG_EXECUTE_NOT_LOGGED | DEBUG_EXECUTE_NO_REPEAT);

          // Revert the mask and callback so we can do some output
          pDebugClient->SetOutputMask(ulOriginalMask);
          pDebugClient->SetOutputCallbacks(pOriginalCallback);

          // Display 'vertarget'
          pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_CLIENTS,  
            DEBUG_OUTPUT_NORMAL, "[Normal]\n%s\n", pOutputCallbacks->
            BufferNormal() ? pOutputCallbacks->BufferNormal() : "");
          pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_CLIENTS, 
            DEBUG_OUTPUT_ERROR , "[Error] \n%s\n", pOutputCallbacks->
            BufferError() ? pOutputCallbacks->BufferError() : "");

          // Go back to our mask and callback so we can do the private callback
          pDebugClient->SetOutputMask(pOutputCallbacks->SupportedMask());
          pDebugClient->SetOutputCallbacks(pOutputCallbacks);

          // Clear the callback buffers
          pOutputCallbacks->Clear();

          // Execute the passed command
          pDebugControl->Execute(DEBUG_OUTCTL_THIS_CLIENT | 
            DEBUG_OUTCTL_NOT_LOGGED, args, DEBUG_EXECUTE_NOT_LOGGED |  
            DEBUG_EXECUTE_NO_REPEAT);

          // Revert the mask and callback so we can do some output
          pDebugClient->SetOutputMask(ulOriginalMask);
          pDebugClient->SetOutputCallbacks(pOriginalCallback);

          // Display the passed command
          pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_CLIENTS, 
            DEBUG_OUTPUT_NORMAL, "[Normal]\n%s\n", pOutputCallbacks->
            BufferNormal() ? pOutputCallbacks->BufferNormal() : "");
          pDebugControl->ControlledOutput(DEBUG_OUTCTL_ALL_CLIENTS, 
            DEBUG_OUTPUT_ERROR , "[Error] \n%s\n", pOutputCallbacks->
            BufferError() ?  pOutputCallbacks->BufferError() : "");
        }

        // Revert the mask (again) for the case 
        // where the mask was set but the callback wasn't
        pDebugClient->SetOutputMask(ulOriginalMask);
        pOutputCallbacks->Release();
      }
    }
    pDebugControl->Release();
  }
  return S_OK;
}

It implements exactly the same functionality as !commandecho. To be able to call Execute and ControlledOutput, I frequently needed to toggle the output mask and output callback values between the passed values and my values to get the desired outcome. Note that the ControlledOutput is passed the DEBUG_OUTCTL_ALL_CLIENTS output control in this scenario, as the associated client is now a desired output target. Additionally, you’ll note that I only implemented (for brevity) error handling on the initial calls.

In most cases, the reuse of a client can become quite cumbersome; this approach should be used with caution.

What to Execute?

The major reason I use this private output technique is to implement DML versions of TEXT-based commands, where I have no knowledge of their internals. In particular, I’ve “value-added” some of the SOS debugger commands with hyperlinks. Going through how I parse the output to decide what to mark up is beyond the scope of this article. I will, however, talk about how I determine the correct command to execute to produce the output I require for the parsing.

The enumeration of loaded debugger extensions and their associated abilities isn’t part of the Debugger API. To make matters worse, the same debugger extension can be loaded multiple times from the same location on disk. If you run the test_windbg.cmd script with the .load myext and .chain commands, you’ll see an example of this situation. The MyExt extension is loaded twice, once as myext and again as myext.dll:

0:000> .load myext
0:000> .chain
Extension DLL search Path:

C:\Debuggers_x86\WINXP;C:\Debuggers_x86\winext;...
Extension DLL chain:

myext: image 6.1.7600.16385, API 1.0.0, built Sat Jan 29 22:00:48 2011
   [path: C:\Debuggers_x86\myext.dll]

myext.dll: image 6.1.7600.16385, API 1.0.0, built Sat Jan 29 22:00:48 2011
   [path: C:\Debuggers_x86\myext.dll]

dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 12:08:26 2010
   [path: C:\Debuggers_x86\dbghelp.dll]
   ...

When calling an extension command for parsing, you want to be sure that you’re calling the intended command, not a command with the same name on another extension (or another instance).

If you fully qualify a command, the engine simply executes the command specified. But when you don’t fully qualify a command, the engine has to do a search. The engine looks at the export table of each extension in the load order and executes the first extension with a matching command name. The .extmatch command displays the current search results for the given command. The .extmatch command supports wildcards of the 
extension name and the command via the “/e” switch, as shown here:

0:000> .extmatch callbackecho
!myext.callbackecho
!myext.dll.callbackecho
0:000> .extmatch /e *myext.dll* *back*
!myext.dll.callbackecho
!myext.dll.callbacktoggle

When you load the SOS extension via a .loadby sos.dll mscorwks call, a fully qualified path will be registered for it; that is, it won’t be known as “!sos.dll.<command>.” In the following example, I ran the Test03 example application on Windows 7 x64 and attached the debugger to it (F6). The SOS extension was loaded from the mscorwks folder (C:\Windows\Microsoft.NET\Framework64\v2.0.50727):

0:004> .loadby sos.dll mscorwks
0:004> .chain
Extension DLL search Path:
  C:\debuggers\WINXP;C:\debuggers\winext;...
Extension DLL chain:
  C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.dll:
    image 2.0.50727.4952, API 1.0.0, built Thu May 13 05:15:18 2010
    [path: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.dll]
    dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 12:15:44 2010
    [path: C:\debuggers\dbghelp.dll]
    ...

The only way to safely call an SOS command is to use full qualification. So, when leveraging the SOS extension (or any other extension), there are two facts to determine:

  • Is the SOS extension loaded?
  • Where is it loaded?

Both can be determined via an extension match (.extmatch /e *sos.dll* <command>). In my LoadSOS function (part of Example08; the code isn’t listed here) I first look for the command in the versioned location (\v2.0.50727\sos.dll.<command>). If I can’t find the versioned command, I fall back to looking for the command without a version (\sos.dll.<command>). If I can now find it, I still return success, but as S_FALSE (as opposed to S_OK). It’s up to the caller of LoadSOS to determine whether it will take an incorrect parsing risk.

If I can’t find an SOS version of the command, I assume it isn’t loaded (or it’s loaded, but not in a way that I’m happy with) and do a private “.loadby.” If this doesn’t generate any errors, I repeat the search. If I can’t load SOS or can’t find my command, I return failure (E_FAIL).

Once I have the command, I execute it, rewrite it (for example, ParseDSO) and output the DML version (refer to Example08 in the code download).

In my DML wrappers of SOS commands, I do this check prior to executing a command to be sure that SOS hasn’t been unloaded between calls and that the required command exists. Figures 9 and 10 show my !dsox command in action with x86 and x64 targets.

Test03 x86 !dsox Wrapping !dumpstackobjects

Figure 9 Test03 x86 !dsox Wrapping !dumpstackobjects

Test03 x64 !dsox Wrapping !dumpstackobjects

Figure 10 Test03 x64 !dsox Wrapping !dumpstackobjects

The commands add a !dumpobj hyperlink to each address listed in !dumpstackobjects; the code for !dsox is in Example08.

Break!

With a little bit of infrastructure code, it’s possible to harness the logic of any built-in or extension command. My parsing of the SOS extension is about adding DML, but it could’ve easily been about the gathering of managed-class information. I didn’t have to know anything about the internal structures of the CLR to achieve my goal. I just had to know how to navigate to the data I wanted to display.

Debugger Extensions simplify debug analysis through the automation of data retrieval and the enrichment of display. I hope this series has inspired you to make your own extensions that simplify the debugging you do.

If you’re just interested in debugging and want to learn more, you should check out the Advanced Windows Debugging and Troubleshooting blog (blogs.msdn.com/b/ntdebugging)—it has lots of training and case study articles to read.


Andrew Richards  is a Microsoft senior escalation engineer for Windows OEM. He has a passion for support tools and is continually creating debugger extensions and applications that simplify the job of support engineers.

Thanks to the following technical expert for reviewing this article: Drew Bliss