This article may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist. To maintain the flow of the article, we've left these URLs in the text, but disabled the links.

A Tale of Real-world Debugging

Matt Pietrek

T his month, I'd like to tell a tale of mistakes, ignorance, and outright lies. No, not the current presidential campaign, but my first faltering steps into Active Server Page (ASP) programming. What started out as an experiment with some simple code in VBScript rapidly deteriorated to using a system-level debugger to poke around at data structures internal to Windows NT®. This was not supposed to happen.       Why tell this tale? There are several reasons. The first is to show how all of these technologies layered atop each other can easily get out of hand. This is especially true when you don't have good visibility through all the layers. Second, I want to illustrate how I worked through a real-world debugging problem that looked intractable. Finally, it's an opportunity to look back at what I did, and see how I could have improved the process.       Currently I'm not much of a Web/VBScript/distributed computing kind of guy, but I decided it was time to get my feet wet. After installing Microsoft® Internet Information Services (IIS) on a box running Windows® 2000, and creating the obligatory "Hello World" Web page in static HTML, I was antsy to do some real ASP programming. While not completely unfamiliar with Visual Basic®, I had never used VBScript before. I wanted to write something that used live data, wasn't too complicated, and could be written in VBScript and ASP so I could use a browser to access the data. Ken Ramirez's May 2000 MSDN® Magazine article, "Virtual Source Code Control Systems: Promoting and Managing Projects Using Visual SourceSafe®," provided the seed for my idea.       My goal was to first get the code working in Visual Basic, then port it to IIS. In relatively short order, I had a 25-line Visual Basic-based program using Microsoft Visual SourceSafe automation to open and read a Visual SourceSafe database on my company network. The next step was to pour the code into ASP and show the results in the browser, rather than in a Visual Basic listbox. Wanting to travel light, I wrote my ASP pages in my code editor, copied them to the appropriate IIS directory, fired up the browser, and promptly received syntax error messages.       Fixing the initial errors was relatively painless. It mostly involved changing all my Dim foo as bar statements into just Dim foo. I feel naked without the strong typing of C++, but if you want to use VBScript, you have to leave that security blanket behind. In practically no time, I had an ASP page beginning to open the Visual SourceSafe database like this:

Dim db 
set db = Server.CreateObject( "SourceSafe" )

However, the next line gave me problems:

db.Open "R:\SourceDir\MyProject\srcsafe.ini", "MATT"

      This is where the descent into madness began. When viewed in the browser, I got the following error message: "An exception of type 'SourceSafe: The SourceSafe database path MATT does not exist. Please select another database.' was not handled."
      Strange, I thought. The first parameter specifies the database to open, and "MATT" is the user login name, not a database path. The message just didn't make sense. After poking around trying different combinations of parameters for a short while, no solution was forthcoming.
      At this point, I recalled reading about minor programming differences between standard VBScript and VBScript in ASP. Why not try my code in normal VBScript? A good idea, but I was clueless as to where to find a program to try my VBScript code with.
      Being stubborn, and trying to avoid reading too much documentation (who has time?), I first tried using Windows Script Host (WSH) by putting my code in a .wsh file. The error messages I got back were unintelligible, and somehow correlated with the first character of the file name I used for the .wsh file. I decided that the WSH probably wasn't a fruitful path to take.
      Finally relenting to patiently searching the MSDN Online site, I learned that any of the Microsoft Office applications is a perfectly fine VBScript programming environment. I had read this before, but I had round-filed it in my brain, thinking that I'd never actually write VBScript code for Microsoft Office.
      With an Alice in Wonderland-like feeling, I brought up the Visual Basic editor inside of Outlook® (Tools | Macro | Visual Basic Editor). Pasting my original working Visual Basic code into the Visual Basic editor from Outlook, I quickly had the code connecting to Visual SourceSafe and navigating through projects. Hmmm... My code worked in Visual Basic, it worked in VBScript from Outlook, but I hit a brick wall when running the same code from within IIS.
      Looking back, my enthusiasm to get something running quickly got the better of me. I would have saved time in the long run by doing less thrashing about in trial and error programming. It was at this point that I realized I should have bitten the bullet earlier and learned how to use the proper toolâ€"in this case, Visual InterDev®.
      Resigned to the fact that the problem only occurred when my code ran under IIS, my hope was that the Visual InterDev debugger could give me better information about the strange Visual SourceSafe error message. After installing Visual InterDev, I quickly created a new project, and added my ASP page to it. Whoosh! Visual InterDev disappeared completely. I tried it again and Visual InterDev disappeared again. It's strange how when a programmer bangs his head into a wall, he almost immediately bangs it again, thinking the wall will yield the second, third, or fourth time.
      Thinking there was something seriously wrong with my code, I created a brand new Visual InterDev project and added a blank ASP page. The same problem occurred: Visual InterDev still just vanished. The next logical step was to try installing the latest Visual Studio® service pack. Still no luck, and I had no Plan B to fall back on.
      I did a search of MSDN Online to see if this was a known problem. I found many Knowledge Base articles about Visual InterDev crashes, but nothing that fixed my particular problem. Only later did I discover that I had long ago installed a prerelease version of Microsoft Internet Explorer 5.5. This version uses a different MSHTML.DLL than Windows 2000, and Visual InterDev was faulting in MSHTML.DLL.
      Since uninstalling Internet Explorer 5.5 wasn't an option (another Microsoft beta product required it), the next logical step was to install Visual InterDev on my other machine. On this machine (also running Windows 2000, but without IIS installed), Visual InterDev worked great. I was good to go with IIS on one machine and Visual InterDev on the other, successfully reading and writing ASP files on the IIS machine. However, in using two machines, I had unwittingly set myself up for the torture known as remote debugging.

Into the Abyss

      Thinking I was finally ready to debug my ASP script, I set a breakpoint and hit F5. Almost immediately, I was presented with a dialog box from Visual InterDev telling me that it wasn't able to find the server to start debugging. That's strange, I thought. Somehow Visual InterDev is able to find the server when it writes the ASP files to it. So I went back to the documentation, where I determined that I needed to enable remote debugging.
      From MSDN Online, I learned that I had to install an entirely new set of Visual Studio Enterprise Edition features: the BackOffice® Server 4.5 Developer Edition tools. After the Visual Studio setup finished, I followed a nine-step set of instructions to configure DCOM for remote debugging. With that done, I followed another set of steps to enable manual debugging of ASP pages.
      Finally I was finished configuring things (or so I naively thought) and I switched my attention back to the machine running Visual InterDev. Again, I hit F5 to run, and sure enough I got the same error message about not being able to find the server for debugging. Just for kicks, I rebooted the IIS machine. Still no debugging after the reboot.
      Running out of things to try on the IIS machine, I switched back to the machine running Visual InterDev. I started fumbling around, looking for anything to try. Through some strange twist of fate, I unknowingly did something that caused the machine to reboot spontaneously. This was the best luck I had all day, although I was disappointed that I had finally crashed Windows 2000, which is normally rock solid for me. Miraculously, after the machine running Visual InterDev restarted, it was able to connect to the IIS machine for debugging. Sometimes you just accept good fortune without asking too many questions.
      With my new ability to finally debug my VBScript in an ASP page, I promptly tried to step over the line of code that was now my obsession:

db.Open "R:\SourceDir\MyProject\srcsafe.ini", "MATT"

As in my first attempt, I received the same error message, but this time it was nicely rendered in a dialog box (see Figure 1), rather than in HTML via the browser. Much more helpful. So much for the extra debugging power of Visual InterDev, at least in this case.

Figure 1Database Connection Error

      Doubting my sanity at this point, I went back to Ken Ramirez's article and substituted his VSSDatabase::Open call for mine, but substituted my values. Still no luck. However, I did notice that Ken's code used a srcsafe.ini file from a local hard drive, while I was using a srcsafe.ini file on the network. On a lark, I changed my code to use a local copy of srcsafe.ini, and lo and behold, it worked! The local Visual SourceSafe database wasn't of much use to me, but at least I had another data point to work from.
      Thinking that perhaps IIS might be doing something unusual with file names, I tried replacing my original network srcsafe.ini path with the UNC version (that is, \\machine\path, rather than R:\<path>). This didn't pan out.
      At this point I began wondering if the problem had something to do with accounts and security in Windows 2000. I had come across the anonymous user account that IIS uses, and figured it had some sort of restricted access to resources on the network. Still, the Visual SourceSafe error message didn't directly indict Windows 2000 security as an issue. The question became how to tell if the IIS security credentials were truly the cause of the problem.

Bringing in the Heavy Artillery

      For people who live and breathe debuggers, a debugger that won't let you go where you want is almost more frustrating than no debugger at all. The Visual InterDev debugger works fine for script-level debugging, but when a script calls a COM method, you just can't chase the rabbit down the hole. I knew that the ASP page was making a call to the VSSDatabase::Open method, that the method was failing for some reason, but that I couldn't step through the code with Visual InterDev to see why.
      If I was going to figure this out, I needed to use a regular debugger such as WinDBG or the Visual C++® debugger on the process that runs the script code. In theory, I could step through the Visual SourceSafe code that implements the VSSDatabase::Open method to locate where something went wrong. In practice, it's not that simple, for two reasons.
      First, it's not at all clear where I'd set a breakpoint. There is no obvious spot where the script leaves off and the COM object it's calling begins. Second, which process would I attach to? There were four separate copies of DLLHOST.EXE loaded, any one of which could be the one running my ASP page.
      Faced with this problem, my initial response was to go nuclear. I figured that the VSSDatabase::Open method had to be opening a file. As such, I figured that a system-level debugger that operates across all processes could lead me quickly to the code. Since Compuware's SoftIce can set a systemwide breakpoint, I plunked a breakpoint on the CreateFileA and CreateFileW entry points in KERNEL32.DLL. Later I'll explain a better approach that doesn't involve SoftIce (which is made by my company), but stay with me as I explain the path I went down.
      Almost immediately upon setting the breakpoints on CreateFile, they began to be hit. This wasn't good, since I wasn't exercising the ASP code at that moment. Some other process was calling CreateFile repeatedly. I tried using some conditional breakpoints to stop the noninteresting breakpoints, but they were ineffective.
      A better approach was to use the Visual InterDev debugger and SoftIce together. The idea is to let the high-level debugger (Visual InterDev) get close to the problem, and then switch to SoftIce for the deep drilling. To do this, I disabled my breakpoint in SoftIce and set a breakpoint on the db.Open call in the Visual InterDev debugger. Next, I used the browser's Refresh button to reexecute the ASP script.
      Upon hitting the breakpoint in the script code, I popped into SoftIce, reenabled my breakpoints on the CreateFile APIs, and then told Visual InterDev to step over the call. According to my theory, the VSSDatabase::Open code should have called CreateFileA, and SoftIce should have stopped at the breakpoint. In reality, the breakpoint didn't trigger and Visual InterDev merrily told me that "MATT" was not a valid Visual SourceSafe database. (As if I didn't know this after the 150th time.) Ackkkk! Now what?
      Faced with the prospect of punting or figuring out where the VSSDatabase::Open code resided, I chose the latter. During the obligatory head-clearing break to get a soda, I connected the dots to realize that the Visual SourceSafe DLL (SSAPI.DLL) also contains a type library that describes the Visual SourceSafe automation model.
      In my March 1999 article in MSJ, "Improve Your Debugging by Generating Symbols from COM Type Libraries", I presented the CoClassSyms program. CoClassSyms loads an in-proc server DLL (such as SSAPI.DLL) and synthesizes rudimentary debug symbols for it. It's able to do this by creating instances of the exposed CoClass objects, and then examining their vtable entries and the corresponding method names in the type library. After running CoClassSyms on SSAPI.DLL, I had SSAPI.DBG, which had symbols for the IVSSDatabase interface methods.
      Now at least I had a fighting chance of setting a breakpoint reasonably close to where the problem was occurring. While the symbols I built weren't complete by any means, they did include the all-important VSSDatabase::Open method. Loading SSAPI.DBG into SoftIce, I set a breakpoint on VSSDatabase::Open and hit the Refresh button in my browser so that the ASP code would execute. Alas, my breakpoint didn't go off. Something was definitely not working the way it was supposed to.

The Debugger Morass

      It's at this point that knowledge of my tools saved the day. Due to the way SoftIce manages breakpoints, the breakpoint was not actually being set in the process that was using SSAPI.DLL. In order to make SoftIce set the breakpoint properly, I needed to identify exactly which process I wanted the breakpoint in. Getting the process ID (PID) for this process turned out to be another can of worms.
      When I had set up IIS and Visual InterDev for remote debugging, I had to select the IIS option that runs each IIS application in a separate host process, rather than in the main IIS address space. On Windows 2000, this host process is called DLLHOST.EXE. This is the same host executable that COM+ applications run in.
      My problem was that there were four instances of DLLHOST.EXE running on my machine. One of them was running my ASP code and was making the calls to SSAPI.DLL. The question was, which one? I tried snooping around in the Component Services snap-in of the Microsoft Management Console, but it wouldn't give me a process ID for any of the DLLHOST.EXE instances running on my machine.
      Since the straightforward approach wasn't working, I tried the reverse angle. I knew that SSAPI.DLL was loaded in one of the DLLHOST instances. The ModuleList program presented in Under the Hood in the September 1998 issue of MSJ attempts to show a list of all loaded DLLs in the system, and for each DLL it lists the processes that are using it. Unfortunately, ModuleList didn't show SSAPI.DLL as being loaded or used at all.
      The problem turned out to be that the DLLHOST instance was running in a different account than the interactive user. As such, ModuleList wasn't able to open a process handle for the DLLHOST instance to scan its module list. See Win32® Q&A in the March 1998 issue of MSJ for tons of details on this issue. In hindsight, a program such as HandleEx from SysInternals would have found the module, since it uses a different way of accessing the list of loaded DLLs.
      Finally, I used brute force to find the correct DLLHOST instance. While stopped in SoftIce, I listed all of the processes and used another command to switch the active memory context to each DLLHOST instance. In each context, I listed the loaded DLLs and when I saw SSAPI.DLL, I knew I had the right context. Once in the right memory context, SoftIce had no problem setting the breakpoint, and I hit the breakpoint upon reexecuting the ASP .
      Before telling you what I found, I'd like to point out how my own haste and stubbornness made the debugging process much slower. Once I had my mind set on using SoftIce, I ignored the possibility of using WinDBG or the Visual C++ (MSDEV.EXE) debugger. In the same spot where you tell IIS to run your ASP code in a separate process (that is, DLLHOST.EXE), you can also tell IIS to launch the separate process under the control of a debugger. By selecting this option and restarting IIS, I would have ended up in WinDBG or Visual Studio immediately and attached to the correct process rather than in my goose chase, finding the correct process for use with SoftIce.
      In my defense, both WinDBG and MSDEV can be finicky about loading .DBG files, and they may not have liked my SSAPI.DBG. However, in a pinch, I could have dumped the contents of SSAPI.DBG to obtain the logical address of VSSDatabase::Open. I then would have converted this logical address to a linear address and set a breakpoint on that address in WinDBG or MSDEV. As a side benefit, both of these debuggers can automatically load the exports for any DLL in the target process. In the end, I needed both the exports information and my SSAPI.DBG symbols to understand what I was seeing as I stepped through the VSSDatabase::Open code.

Hunting Down the Problem

      With my breakpoint on VSSDatabase::Open being hit successfully, and with symbols and exports loaded, I was finally ready to step through the SSAPI.DLL code to find the problem. Remember, not only was VSSDatabase::Open returning a nonsensical error message, I also wasn't seeing the call to CreateFile that I was expecting. On top of that, I wasn't even stepping through source code. It was just me, a disassembler window, and a small set of symbols.
      Rather than trying to slowly step through the assembly code and understand everything, I started by stepping through the code at a high level. Instead of stepping into each CALL made by VSSDatabase::Open, I stepped over the calls. After each CALL I stepped over, I looked at the return value in the EAX register to see if it looked like an error code. I also looked at the instructions following the CALL instruction. If they continued along the main path of execution, I figured the CALL was probably successful.
      On the other hand, if the instructions following the call branched to code that quickly exited from VSSDatabase::Open, I figured the CALL had failed. Upon finding the top-level CALL that failed, I set a breakpoint on it and restarted my ASP code, causing my new breakpoint to be hit. This time, though, I stepped into the function and began looking for where that function was encountering an error and returning to its caller.
      In many cases, I was lucky because the functions I was stepping into were exported, and the debugger could show me their names. This gave me a little more confidence in my ability to guess what the code was doing, and whether the error condition had been triggered. Nonetheless, finding the error condition wasn't a surefire process that's easily described. There can be many false positives along the way. A lot of my luck stemmed from gut feelings based on years of doing this sort of hunting.
      By following the steps I described to deeper and deeper levels within SSAPI.DLL, I was finally able to find a Win32 API call that I knew had failed, but which I would have expected to have succeeded. The API that was failing was GetFileAttributes. I verified that this wasn't a red herring by looking at the file name parameter passed to GetFileAttributes. The file name was identical to what I was passing to VSSDatabase::Openâ€"that is, the name of my SRCSAFE.INI file on the network drive.
      Once I knew that GetFileAttributes was failing, it was easy enough to see how the remainder of the executed SSAPI.DLL code quickly exited all the way out to the caller of VSSDatabase::Open. Nonetheless, it wasn't obvious why GetFileAttributes was failing. My theory was that it had to do with the security attributes that DLLHOST.EXE was running with. How could I verify that, though?
      When GetFileAttributes fails, it sets the LastError code. Unfortunately, the SSAPI.DLL code doesn't bother to call GetLastError. The good news is that it's easy enough to work around it. I temporarily set the CPU window to the GetLastError function in KERNEL32.DLL, which was all of three instructions:

MOV EAX,FS:[18h]
MOV EAX,DWORD PTR [EAX+34h]
RET 

The GetLastError code grabs the DWORD at offset 0x34 in the thread environment block pointed to by FS:[18h]. It was easy enough to read this DWORD value using SoftIce commands. This is how the Visual C++ debugger gets the ERR pseudoregister value that can be seen in the Watch and QuickWatch windows.       I found that GetFileAttributes was failing with a GetLastError code of 5. Using the Error Lookup tool (ERRLOOK.EXE) that comes with Visual Studio, I quickly determined that a last error code of 5 means Access is denied. This confirmed my hypothesis that the problem was access rights-related, rather than an incorrectly specified parameter. It also meant that the error message from SSAPI.DLL was completely out to lunch.       Eventually, I got around the access rights problem by having IIS start my DLLHOST instance under my interactive user account. This was further evidence that my original hypothesis about account security rights was correct. Yes, this creates a huge, gaping security hole, and if I intended to make a production program out of this, I would find a more secure way of accessing the Visual SourceSafe database.       The lessons learned? When you hit a brick wall, slow down and think about the problem. Consider what the possible problems could be, and how you can prove or disprove them. Also, consider what tools you have available. Perhaps one tool or technique won't do everything, but several tools or techniques creatively combined can give you enough of a toehold so that another tool can be used. Finally, be persistent, but not to the point where you blind yourself to other approaches.

Matt Pietrek does advanced research for the NuMega Labs of Compuware Corporation, and is the author of several books. His Web site, at https://www.wheaty.net, has a FAQ page and information on previous columns and articles. You can reach Matt at matt@wheaty.net.

From the September 2000 issue of MSDN Magazine.