Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
.gif)
.gif)
Windows 2000 and LDR Messages, A COM Symbol Engine, Finding Bloated Functions, and More
John Robbins
Code for this article:Bugslayer0400.exe (53KB)
| I |
n this month's column, I'll start out with a look at some Bugslayer issues in Windows® 2000. After that, I'll turn to something many people have asked about: a COM-based front-end to the IMAGEHLP.DLL/DBGHELP.DLL symbol engine.
I'm also going to discuss a little program that kills two birds with one small program. Back in the October 1999 issue of Microsoft Systems Journal, I discussed reading MAP files (https://www.microsoft.com/msj/1099/bugslayer/bugslayer1099.htm). Based on the huge number of e-mails I received, many people were happy to finally learn how to read them. Several readers also asked about what they could do to generate a MAP file for software they have already released.
In addition, reader Barry Hart challenged Jeffrey Richter, Paul DiLascia, and me to develop a program that tells you how much space your program takes up in memory. While you can use the Process Walker from the Platform SDK to see what memory pages belong to which module and their free, reserved, or committed state, he wanted to look one level lower. Specifically, Barry wanted to know how big individual functions were in memory. The size of your functions can change dramatically when the magical "invisible" inlines, especially with STL, start expanding in place. Jeffrey, Paul, and I all immediately thought along the same lines, but the implementation turned out to be much easier than expected.
The little program I developed, PDB2MAP, will not only give you a better MAP file, but it will tell you where those bloated functions are too!
Windows 2000 and Missing LDR Messages
By the time you read this, Microsoft will have started delivering Windows 2000 to the masses. I want to point out a couple of items that changed in Windows 2000. The first is that the operating system loader no longer reports when the operating system is relocating DLLs in your address space. Instead of the easy-to-see message, you now have to figure out manually if you have collisions.
For those readers who might not know what I am talking about, here's a brief recap. Each DLL has a default load address associated with it. When a second DLL wants to load at an address where another DLL is already loaded, the operating system must relocate the second DLL to another spot in memory. While that doesn't sound too bad, longtime readers of the Bugslayer column take relocations very seriously.
The first problem with relocations is that the operating system will have to run through your DLL's code section and update each offset that references memory so the memory references are correct. Needless to say, this can add considerably to your DLL load time. The second problem is that relocated DLLs take up more room in memory and increase the size of your working set, thus making your app run slower. Finally, and most importantly, different machines will put the relocated DLL at different locations, so it becomes much harder to figure out where your app crashed.
If you had a relocation while running under a debugger on Windows NT® 4.0, you would see a string like the following in the Output window.
LDR Dll foo.dll base 10000000relocated due to collision with bar.dll
But on Windows 2000 you won't get that warning. Consequently, you will have to whip out your copy of DUMPBIN.EXE and look at the module's base addresses manually. The magic command line is
DUMPBIN /HEADERS <module>
Figure 1 shows the first half of a dump. Nine lines below OPTIONAL HEADER VALUES, look for the line that says "image base." To the left is the load address for your module. Check each of the modules in your program to ensure that none of the preferred load addresses are the same. To rectify conflicting addresses, see the Bugslayer column in the April 1998 issue of MSJ (https://www.microsoft.com/msj/0498/bugslayer0498.htm).
I can see the point in removing the message because it comes through a call to OutputDebugString, which can slow down a processâ€"and those LDR messages are only useful a small portion of the time. However, this is one of those cases where Microsoft should have left the call in, but with an option to turn it on.
The IMAGEHLP.DLL/DBGHELP.DLL Saga
In the Bugslayer column in the January 2000 issue of MSJ (https://www.microsoft.com/msj/0100/bugslayer/bugslayer0100.asp), I thought I finally put to rest the question of what was happening with the IMAGEHLP.DLL/DBGHELP.DLL symbol engine. I have to apologize because I made an assumption. I was using Windows 2000 RC2 and assumed that Microsoft had finalized the operating system and was not going to make any other major changes. How was I to know that Microsoft would make a fairly major change to a system DLL between RC2 and RC3?
Consequently, the latest story in the DBGHELP.DLL sagaâ€"at least until I get the actual final bitsâ€"is that DBGHELP.DLL is all you need for symbol lookup. (Microsoft used to require MSDBI.DLL to look up symbols in PDB files.) The even better news is that with DBGHELP.DLL holding all the symbol reading code, you can now distribute symbol lookup with your application, and it works on all other Win32®-based operating systems as well!
Since you should be using DBGHELP.DLL instead of IMAGEHLP.DLL, all of the code I do from now on in the Bugslayer column will use DBGHELP.DLL directly. IMAGEHLP.DLL still exports the symbol engine API, but just forward to DBGHELP.DLL. To use DBGHELP.DLL you need to link against DBGHELP.LIB, which is available in the freely downloadable Platform SDK. Since the Platform SDK holds not only the latest and greatest headers and libraries, as well as excellent tools such as WinDBG, you should always have the Platform SDK installed.
The COM Symbol Engine Saga
Originally it was three birds I planned to kill with this program, but I ran into a conundrum. A few columns ago I asked if readers were interested in having a COM-based symbol engine, and many of you said you would be. However, things didn't work out as planned.
I want to share some of the issues I stumbled into during planning. I think a COM front end to the symbol engine API can be done, but I was not happy with the way things were turning out. I thought I would tell you what I found to get your input.
My first thought was, what is a symbol table other than a specialized database? At first I thought it would be nice to build an OLE DB provider interface since that would be the ultimate in ease of use. However, never having worked with any form of database in my career, I was going to have to learn quite a bit to get up to speed on OLE DB interfaces. Seeing that it was going to take a few months to get introductory databases under my belt, I didn't think everyone would wait that long. While I still think it might be the way to go, I just wasn't knowledgeable enough to do a credible job in time to get this column done. Consequently, I decided to look at building my own custom interface.
At first glance, it looks like a COM symbol engine will be a piece of cake because COM methods would map directly to many functions. For example, the function to load a module into the symbol engine, SymLoadModule, takes a few parameters and returns a Boolean value indicating success or failure. While many of the symbol engine functions take variable-length structures and other structures that would not translate directly into a COM-safe version, the problem is certainly not insurmountable.
Another drawback is that all of the symbol engine interfaces are strictly ANSI characters, so I would have to do a bunch of string conversions to BSTRs. Again, not a big deal. It is just like the conversions Windows 2000 makes when you are running an ANSI app. The downside is that any COM-based symbol engine would run quite a bit slower than using the straight C version.
The part that really threw me for a loop was when I started thinking about the symbol enumeration. Unfortunately, instead of having a simple GetFirstSymbol/GetNextSymbol approach, which would allow easy enumeration as well as enumeration control, the DBGHELP.DLL symbol engine relies on a callback function approach. In other words, to enumerate the symbols, you always have to start with the first one and go to the end unless you abort early by returning FALSE in the enumeration function. In addition to symbol enumeration issuesâ€"as you saw in the LIMODS utility from my June 1999 MSJ column (https://www.microsoft.com/msj/0699/bugslayer/bugslayer0699.htm)â€"there is no clean way to enumerate source files and lines as well.
At about this point I started realizing that building a COM front end was not going to be as easy a task as I thought it would be. After pondering it a bit, I started looking at some possible solutions. My first inclination was to do the proper thing and build an interface that conforms to the IEnum* enumerator interface, thus completely hiding the callback inside the COM symbol engine object. This way, using the COM symbol engine interface would be a piece of cake because the COM interface could implement the GetFirstSymbol and GetNextSymbol methods.
While this is a good idea in theory, the reality was a little tougher. Since the straight C interface requires the callback, I was going to have to implement the data hiding in one of two ways. In the first approach, I could have kept the last enumerated symbol and, when asked for the next symbol, I would call the C interface to enumerate up to the next one. Since most binaries have thousands of symbols, this would be horrifically slow.
To speed things up, I would just have to do the enumeration once and store all the symbols myself. But, I would be duplicating the symbol data in memory, and with thousands of symbols for a module, the memory requirements would grow exponentially. As much as I would like to correctly implement the symbol enumeration code, it was too resource-intensive.
I'll have to be less of a purist and bend a little on the enumeration interface. I could simply have a separate interface, ISymbolEngineCallback, that I pass to my enumeration function, which I would call as part of the internal SymEnumerateSymbols callback processing. This would mostly lock the interface into an in-process COM DLL and lose the ability to remote the interface easily.
Is it worth having a COM front end that isn't a clean COM model? Can you think of a better design? Am I concerned about the wrong things? Let me know, and I will develop an implementation based on everyone's thoughts.
PDB2MAP
As I mentioned in the introduction, PDB2MAP takes care of two things: it gives you a way to produce a MAP-like file for a binary after the binary was built, and it solves Barry Hart's challenge to figure out a way to find the size of each function in your binary. Other than a nasty performance problem I had to solve, the solution for both goals was rather easy.
When writing out the MAP-type file I wanted to have a format that was easier to read than the standard linker-produced MAP files. Instead of showing offsets based on code sections, I wanted exact addresses associated with each function and line. Your old friend the DBGHELP.DLL symbol engine has all of that information. The only limitation with PDB2MAP is that you need to always build your application with either C7 (CodeView) or PDB debug symbols for all build types, which is something I have been recommending since almost my first Bugslayer column.
Figure 2 shows a partial P2M file produced by PDB2MAP. As you can see, the format is very similar to a MAP file and is easier to read. At the top is the header information, which lists the module name, link time, and preferred load address. The second section shows the public functions sorted in ascending address order. Since the DBGHELP.DLL symbol engine will not return static functions, there is no way to display them with PDB2MAP.
The final section shows the line information for each source file. The format is LINE : ADDRESS in keeping with the MAP file format, though PDB2MAP does not require the odd hexadecimal manipulations to figure out the address. Like a MAP file, PDB2MAP sorts the source file lines in address order, not line order.
If you use PDB2MAP on an application written and compiled in Visual Basic®, note that the line numbers reported in the P2M file correspond to the line numbers including the Visual Basic header information. The Visual Basic editor shows the line numbers starting after the header information. To determine the real line as shown in the P2M file, bring up the file in a text editor.
The only other feature of note in PDB2MAP is the /u switch. By default, PDB2MAP P2M files show the C++ public functions in all their name-decorated glory. However, if you would rather see your function names in a format you can read, use the /u switch to tell PDB2MAP to undecorate all the names.
For you function size aficionados, the public function section shows the function size right next to the address. If you are using inline functions heavily, it is interesting to see how the functions change when you build your application using various compilation settings. At least now you have a tool where you can see the effects easily. Before PDB2MAP, you had to manually calculate function sizes.
PDB2MAP Implementation
For the most part, PDB2MAP was pretty straightforward. You can see from PDB2MAP.CPP, the main file in the project (which you can download from the link at the top of this article), that most portions are easy to understand. I want to mention three things that made the implementation interesting.
The first concerns the size calculations. When Barry Hart proposed the idea to Jeffrey, Paul, and me, we all immediately thought of parsing the MAP file and then figuring out the size from adjacent function addresses. Since I personally avoid parsing like the plague because it's so hard to debug and get right, I realized I could do the same thing using the DBGHELP.DLL symbol engine.
All I needed to do was call SymEnumerateSymbols, enumerate all the symbols into an array, and do the calculation manually. I was all set to do that when I noticed that the function you pass in to enumerate the symbols has a parameter that corresponds to the symbol size. The documentation states: "The size is calculated and is actually a best-guess value. In some cases, the value can be zero."
When I did the enumerations I found that the version of DBGHELP.DLL that ships with Windows 2000 RC3 works like a champ and reports the symbol size. So, I got Barry's challenge feature absolutely free. I love it when a feature takes no work at all.
After I had the public function information going, I turned to the source file line information. Unlike the public functions, there is no clean interface for enumerating source lines. The DBGHELP.DLL symbol engine allows you to enumerate source lines with SymGetSymNext and SymGetSymPrev, but those functions only return the source lines for a small range of the source file.
Back in my June 1999 Bugslayer column where I implemented the LIMODS program that allows you to decide which trace statements you want to see, I ran into the exact same problem. That source line information appears in chunks, which I called ranges. A source file that has inline statements can actually appear spread out in the binary, depending on where different functions expand. The problem is that SymGetSymNext and SymGetSymPrev will only enumerate inside a single range. In other words, they will not jump ranges so you can enumerate all the lines in a source file, no matter where they would normally appear in the final module.
To work around this limitation, I came up with a scheme where I looped through all the functions. With each function I look to see if it has associated line information. If it does, I check to see if I have already processed the source file that function resides in. If I have not seen that source file at all, I add the source file and the entire range to my data structures.
If the source file is already in my data structures, I look to see whether I have the function address and line number already. If I do not have that pair, I assume that it is a new range so I add that range to the existing data structure. After running PDB2MAP on many programs and comparing the P2M and linker-produced MAP files, I am confident that my algorithm works.
One thing I did notice was that the line numbers for global classes are not handled by the DBGHELP.DLL symbol engine. For example, on line 89 of PDB2MAP.CPP, I have the line "CSymbolEngine g_cSym ;" and line 89 never shows up from the symbol engine, so PDB2MAP can't report it. That's the only line information that is different between a MAP file and a P2M file.
After designing and implementing all the algorithms, PDB2MAP ran really slow. I was originally using STL for my data structures, which were rather simple. The public function information was stored in an STL vector array, which was easy and quick as I expected. The problem was how I was storing the line information. I was storing a class that contained two member variables, a string with the source file name, and an array of address/line pairs. What I failed to realize was that the STL find function just does a linear search on the data structures. Since I was searching on a source file's address/line pairs quite a bit, it was taking forever to loop through the data. Clearly, I had to find a better search algorithm.
The other problem was that I used a commercial performance tool to double-check my hypothesis, and noticed many calls to various class constructors. Since I could not use by reference arguments to the templates, there were many constructors called each time I used an STL function or STL copied my data.
The only thing I remembered from my college data structures class was that a binary search beats a linear search every time. I flipped open my trusty STL reference and was immediately stumped. The STL binary_search function returns true if the item is in the data structure, but does not return the data structure. Since I wanted to search on the source file name and get the array of address/line pairs, using binary_search was not going to work.
After a few seconds of pondering, it dawned on me that I just needed a map class with my key being the source file name. After much gnashing of teeth and far too much time spent figuring out what 800-character error messages spewing from the compiler were really trying to say, I realized that I wasn't getting anywhere. I was trying to write a utility that was very fast and maintainable. As I was getting everything to line up with STL, I was making the code far more complicated than it needed to be.
Sometimes, in my quest to use something like STL, I forget that my mission is twofold: to solve the problem in a fast and efficient manner, and to make that solution maintainable for the future. Writing maintainable code is hard because it involves making some trade-offs and predictions about the future. When, in addition you try to write fast code, you start to make life difficult. While some people like to overcomplicate things because they feel smart, I feel smart when my code has a very simple grok factor and is very fast.
The grok factor is when a recent college graduate can read your code and get a solid idea of how it all fits together. In the case of PDB2MAP, I went back to the multiple array system. The main array contains structures that hold the source file name and the array to the address/line pairs. By using these simple arrays, I avoided the extra constructor calls because there is one and only one instance of any piece of data. Second, I could still use the tried and true C run/time qsort and bsearch functions directly on the arrays. Finally, anyone can understand the code. The icing on the cake is that the final version of PDB2MAP is also much faster than a partial version I implemented with STL.
Wrap-up
Now that you have PDB2MAP in your arsenal, you can finally get an idea of how different compiler options affect your function sizes. Not only that, you get a MAP file that is much easier to read.
Da Tips!
Tip 31 While debugging, CTRL+F11 (or ALT+8 with Visual C++® 2.0 keyboard mappings) will toggle the current source view to the Disassembly window. Unlike pressing the Disassembly button on the debug toolbar which opens the view at a random location in the code, CTRL+F11 takes you right to the disassembly for the line that you were viewing in the Source window. To change your keyboard mapping to use another key to jump to the disassembly, the ActivateDisassemblyWindow command can be assigned to the new key. (Tip from Sean Echevarria.)
Tip 32 The Microsoft Knowledge Base (KB) has all sorts of wonderful information, but can be overwhelming. Fortunately, all the KB articles have subcodes associated with them, which help to narrow the search. The codes for each KB article are at the bottom of the article. For example, searching for "ntstop" will yield all KB articles related to STOP messages/blue screens for Windows NT. "How to Query the Microsoft Knowledge Base Using Keywords" Q242450 (https://support.microsoft.com/support/kb/articles/Q242/4/50.asp), gives you tips for searching the Knowledge Base with keywords.
John Robbins is an independent consultant and also teaches Windows debugging courses. He is the author of Debugging Applications (Microsoft Press, 2000). Reach John at https://www.jprobbins.com.
From the April 2000 issue of MSDN Magazine.