Debugging with Symbols

This article provides a high level overview of how to best use symbols in your debugging process. It explains how to use the Microsoft symbol server, and also how to set up and use your own private symbol server. These best practices can help increase your effectiveness and ability to debug issues, even in cases where all the symbols and executable files that are related to a problem are not located on your computer.

• Symbols
• Using Symbols for Debugging
• Getting the Symbols You Need
  • Check if a given DLL or .exe file and PDB in the same folder match
  • Check if all the DLLs and executable files in a set of folders have matching PDBs
  • How symchk Works
• Symbol Servers
• Using the Microsoft Symbol Server
• Getting Symbols Manually
• Setting Up a Symbol Server
• Adding Symbols to a Symbol Server
• Best Practices

Symbols

A number of different types of symbols are available for debugging. They include CodeView symbols, COFF, DBG, SYM, PDB, and even export symbols that are generated from a binary files export table. This white paper discusses only VS.NET and the PDB format symbols, because they are the most recent, preferred format. They are generated by default for projects that are compiled by using Visual Studio.

Generating PDB files for release executables does not affect any optimizations, or significantly alter the size of the generated files. Typically, the only difference is the path, and the file name of the PDB file is embedded in the executable. For this reason, you should always produce PDB files, even if you don't want to ship them with the executable.

PDB files are generated if a project is built by using the /Zi or /ZI (Produce PDB Information) compiler switch, together with the /DEBUG (Generate Debug Info) linker switch. The compiler-generated PDB files are combined and written into a single PDB file that is placed in the same directory as the executable.

By default, PDB files contain the following information:

• Public symbols (typically all functions, static and global variables)
• A list of object files that are responsible for sections of code in the executable
• Frame pointer optimization information (FPO)
• Name and type information for local variables and data structures
• Source file and line number information

If you are concerned about people using the PDB file information to help them reverse engineer your executable, you can also generate stripped PDB files, by using the /PDBSTRIPPED:filename linker option. If you have existing PDB files that you would like to strip private information from, you can use a tool called pdbcopy, which is part of the debugging tools for Windows.

By default, stripped PDB files contain the following information:

• Public symbols (typically only non-static functions and global variables)
• A list of object files that are responsible for sections of code in the executable
• Frame pointer optimization information (FPO)

This is the minimum information that is required to allow reliable debugging. Minimum information also makes it difficult to obtain any additional information about your original source code. Because both a stripped PDB file and a regular PDB file are generated, you can provide the stripped version to users who may need limited debugging abilities, but keep the full PDBs confidential. Note that /PDBSTRIPPED generates a second, smaller PDB file, so make sure that you use the correct PDB file when you generate builds to distribute broadly. For a typical project, a regular PDB may be a few megabytes in size, but a stripped version of the PDB may be only a few hundred kilobytes.

Using Symbols for Debugging

When you are debugging an application that has crashed, the debugger attempts to show you the functions on the stack that led up to the crash. Without a PDB file, the debugger cannot resolve the function names, their parameters, or any local variables that are stored on the stack. If you debug 32-bit executables, there are situations where you cannot even get reliable stack traces without symbols. Sometimes it's possible to look at the raw values on the stack, and work out which values might be return addresses, but these can be easily confused with function references or data.

If functions on the current stack were compiled by using the Omit Frame Pointers (/Oy) optimization, and if symbols are not present, the debugger cannot reliably determine which function called the current function. This is because without the Frame Pointer Optimization (FPO) information that PDBs contain, the debugger cannot rely on the frame pointer register (EBP) to point at the saved previous frame pointer and at the return address of the parent function. Instead, it guesses. Sometimes it gets it right. However, it often gets it wrong, which can be misleading. If you see a warning about missing symbols, or no symbols loaded, as in the following example, do not trust the stack from that point down.

SWPerfTest.exe!TextFunction(... ...)    Line 59    C++
d3dx9d.dll!008829b5()
[Frames below may be incorrect and/or missing, no symbols loaded for d3dx9d.dll]
SWPerfTest.exe!main(int argc=, const char * * argv=)  Line 328 + 0x12 bytes     C++
SWPerfTest.exe!__mainCRTStartup() Line 716 + 0x17 bytes    C
kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes
ntdll.dll!__RtlUserThreadStart@8() + 0x27 bytes

In many cases, it's possible to continue debugging without symbols, because the problem is in a location that has accurate symbols, and you don't need to look at functions further down the call stack. Even if a library that is in your call stack doesn't have PDBs available, as long as they were compiled with frame pointers, the debugger should be able to guess correctly at the parent functions. Starting with Windows XP Service Pack 2, all Windows DLL and executable files are compiled with FPO disabled, because it makes debugging more accurate. Disabling FPO also allows sampling profilers to walk the stack during run-time, with minimal performance impact. On versions of Windows before Windows XP SP2, all operating system binaries require matching symbol files that contain FPO information, to allow accurate debugging and profiling.

If you debug 64-bit native executables, you do not need symbol files to produce valid stack traces, because x64 operating systems and compilers are designed not to require them. However, you still need symbol files to retrieve the function names, call parameters and local variables.

However, some cases are particularly difficult to debug without symbols. For example, if you debug a program for which you built a PDB file, and if you crash in a callback from a function in a DLL that you don't have symbols for, you will not be able to see which function caused the callback, because you will not be able to decode the stack. This frequently happens in third-party libraries, if PDBs are not provided, or in old operating system components, if PDBs are not available. Callbacks often happen during message passing, enumeration, memory allocation, or exception handling. Debugging these functions without an accurate stack can be frustrating.

To reliably debug mini-dumps that are generated on a different computer, or that crashed in code that you do not own, it's important to be able to access all the symbols and binaries for the executables that are referenced in the mini-dump. If the symbols and binaries are available from a symbol server, they are automatically obtained by the debugger. For more information on mini-dumps, see the Crash Dump Analysis white paper.

Getting the Symbols You Need

Visual Studio and other Microsoft debuggers, such as WinDbg, are typically set up to just work if you are building an application and debugging it on your own computer. If you need to give your executable to someone else, if you have multiple versions of a DLL or an .exe file on your computer, or if you want to accurately debug an application that uses Windows or other libraries, such as DirectX, you need to understand how debuggers find and load symbols. The debugger uses either the symbol search path that is specified by the user—which is found in Options\Debugging\Symbols in Visual Studio—or the _NT_SYMBOL_PATH environment variable. Typically, the debugger searches for matching PDBs in the following locations:

• The location that is specified inside the DLL or the executable file.

  If you have built a DLL or an executable file on your computer, by default the linker places the full path and file name of the associated PDB file inside the DLL or the executable file. When you debug, the debugger first checks to see if the symbol file exists in the location that is specified inside the DLL or the executable file. This is helpful, because you always have symbols available for code that you have compiled on your computer.

• PDBs that may be present in the same folder as the DLL or executable file.

• Any local symbol cache folders.

• Any local network file share symbol servers.

• Any Internet symbol servers, such as the Microsoft symbol server.

To make sure that you have all the PDBs that you need for accurate debugging, install the debugging tools for Windows. The 32 and 64 bit versions can be found at Debugging Tools for Windows.

A useful tool that is installed with this package is symchk.exe. It can help to identify missing or incorrect symbols. This tool has a large number of potential command line options. Here are two of the more useful and commonly used ones.

Check if a given DLL or .exe file and PDB in the same folder match

"c:\Program Files\Debugging Tools for Windows\symchk" testing.dll /s .

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

The /s . option tells symchk to look for symbols only in the current folder, and not to look in any symbol servers.

Check if all the DLLs and executable files in a set of folders have matching PDBs

"c:\Program Files\Debugging Tools for Windows\symchk" *.* /r

The /r option sets symchk to recursively traverse through folders, to check that all the executable files have matching PDBs. Without the /s option, symchk uses the current _NT_SYMBOL_PATH to search for symbols on any private or local server, or on the Microsoft symbol servers. The symchk tool searches only for symbols for executable files (.exe, .dll, and similar). You cannot use wild cards search for symbols for non-executable files.

How symchk Works

When the linker generates .dll, executable, and PDB files, it stores identical GUIDs in each file. The GUID is used by tools to determine if a given PDB file matches a DLL or an executable file. If you alter a DLL or an executable file—by using a resource editor or copy protection encoding, or by altering its version information—the GUID is updated and the debugger cannot load the PDB file. For this reason, it's very important to avoid manipulating the DLL or executable file after it is created by the linker.

You can also use the DUMPBIN utility that comes with VS.NET to show the symbol paths that are searched, and to see if symbol files are found that match a given DLL or executable file. For example:

DUMPBIN /PDBPATH:VERBOSE filename.exe

Symbol Servers

A symbol server is a repository for multiple versions of executable and symbol files. It contains either the symbol files themselves, or pointers to the associated symbol files. Debuggers understand how to use symbol servers, and can use them to search for missing or unknown symbols.

DLL and executable files are also available from the Microsoft symbol server. This makes it possible to debug crashes and examine code for operating system files that may not exist on your machine. If a debugger encounters an executable file or a DLL that does not exist on the system that you are using for debugging, it automatically requests both the symbols and a copy of the binary file from the Microsoft symbol servers. This is helpful if you are debugging a component that has many versions—for example, msvcrt.dll—and you need to examine the code for a version that does not exist on your computer. This also helps debug mini-dumps that are generated on an operating system that is different from the system that you are using for debugging.

Microsoft publishes all the PDB files for all operating systems and other redistributed components, such as the DirectX SDK, on its externally accessible symbol server. This makes it easy to debug an application that uses these DLL or executable files. You can use the Microsoft symbol server to resolve symbols, together with any local symbols for components that were built on your computer.

You can set up your computer to use the Microsoft symbol server, which gives you access to all Microsoft symbol files. You can also set up a private symbol server for your company, team or network, which can be used to store multiple older versions of a project you are working on, or to provide a local cache for the symbols that you use from the Microsoft symbol server.

To use a symbol server, specify the search path in an environment variable that is called _NT_SYMBOL_PATH. Debuggers and modern tools, such as WinDbg, NTSD or Visual Studio, automatically use this path to search for symbols.

When a debugger searches for symbols, it first searches locally. Then it looks on symbol servers. When it finds a matching symbol, it transfers the symbol file to your local cache. The symbols for a typical DLL or executable file range from 1 to 100 MB in size. Therefore, if you are debugging a process that includes many DLLs, it can take some time to resolve all the symbols and transfer them to a local cache.

Using the Microsoft Symbol Server

The Microsoft symbol server allows you to obtain all the latest symbols, including symbols for patched or updated files. The Microsoft symbol server is available at https://msdl.microsoft.com/download/symbols.

You can access the symbol server in one of the following ways:

• Enter the server address directly. In Visual Studio, from the Tools menu, choose Options, then choose Debugging, and then choose Symbols.

• Use the environment variable _NT_SYMBOL_PATH. We recommend this method.

  This is used by all debugging tools. It is also used by Visual Studio, and is read and decoded when Visual Studio opens. Therefore, if you change it, you need to restart Visual Studio.

  This environment variable allows you to specify multiple symbol servers—for example, an internal private symbol server. It also allows you to specify a local cache directory to store PDBs for all symbols that you look up from symbol servers, both internally and over the Internet.

The syntax for the _NT_SYMBOL_PATH variable is:

srv*[local cache]*[private symbol server]*https://msdl.microsoft.com/download/symbols

Replace [local cache] with the name of a directory on your computer where you want to store a cache of any symbols used—for example, %SYSTEMROOT%\Symbols, or c:\symbols.

The [private symbol server] is optional. It can point to a symbol server that is located on your network, or it can point to a symbol server that is shared by your team, product group, or company.

To use only the Microsoft symbol server together with a local cache of symbols, to speed up access over the Internet, use the following setting for _NT_SYMBOL_PATH:

srv*c:\symbols*https://msdl.microsoft.com/download/symbols

You can find other options for the _NT_SYMBOL_PATH in the help file that is installed with the Microsoft Debugging Tools for Windows package.

Executables without symbols can increase the time it takes to launch a debugger if you use a symbol server. This is because the debugger queries the symbol server each time it tries to load the executable. For this reason, it is best to always request symbols for all components.

It may not be possible to request symbols for every component—for example, video drivers may have DLLs in your process space, and the required PDB files are available on the Microsoft symbol server. In this case, there is a small delay when you start a debugging session.

To avoid even this small delay, you can run the debugger once, to cache all the symbols locally from the Microsoft symbol server. Then, modify your _NT_SYMBOL_PATH to remove the Microsoft symbol server. Unless the executable files change, checks for executable files that do not have symbols will not require a query over the Internet, because you have local cached copies of all the symbols that you need from the Microsoft symbol server.

Getting Symbols Manually

If you have set up your debugger correctly, it automatically loads any symbols that it requires from your local cache or from a symbol server. If you would like to get the symbols for just a single executable, or for a folder of executables, you can use symchk. For example, if you want to download the symbols for the d3dx9_30.dll file in the Windows System folder into the current directory, you can use the following command:

"c:\Program Files\Debugging Tools for Windows\symchk" c:\Windows\System32\d3dx9_30.dll /oc \.

The symchk tool has many other uses. For details, see symchk /?, or look in the Microsoft Debugging Tools for Windows documentation.

Setting Up a Symbol Server

Setting up a symbol server is very simple. It is useful for the following reasons:

• To save bandwidth, or to speed up symbol resolution for your company, team or product. An internal symbol server on a local file share on your network caches any references to external symbol servers, such as the Microsoft symbol server. A local or internal symbol server can be accessed quickly by many people at the same time. Therefore, it saves bandwidth and the latency that duplicate symbol requests can create.
• To store symbols for old builds, versions or external releases of your application. By storing the symbols for these builds on a symbol server that you can easily access, you can debug crashes and problems in these builds on any computer that has a debugger and a connection to the local symbol server. This is particularly useful if you debug mini-dumps that are generated by executables that you did not build yourself—that is, builds that were generated by another programmer or by a build machine. If the symbols for these builds are stored on your symbol server, you will have reliable and accurate debugging.
• To keep symbols up to date. When components are updated, such as OS components that are modified by Windows Update or by the DirectX SDK, you can still debug by using all the latest symbols.

Setting up a symbol server on your own local network is as simple as creating a file share on a server and giving users full permissions to access the share, to create files and folders. This share should be created on a server operating system, such as Windows Server 2003, so that the number of people who can access the share simultaneously is not limited.

For example, if you set up a file share on \\mainserver\symbols, then the members of your team set the _NT_SYMBOL_PATH to the following:

Srv*c:\symbols*\\mainserver\symbols*https://msdl.microsoft.com/download/symbols

As symbols are retrieved, files and folders appear in the \\mainserver\symbols shared directory, as well as in individual caches, in the c:\symbols directory.

This is typically all that is involved in setting up and using either your own symbol server, or the Microsoft symbol server.

Adding Symbols to a Symbol Server

To add, delete or edit files on a symbol server share, use the symstore.exe tool. This tool is part of the Microsoft Debugging Tools for Windows package. Full documentation on symbol servers, the symstore tool, and indexing symbols is included in the Debugging Tools for Windows package.

You may want to add symbols directly to your own symbol server, as part of a build process, or to make symbols available to your whole team for third-party libraries or tools. The process of adding a symbol to a symbol server file share is called indexing symbols. There are two common ways to index symbols. A symbol file can be copied to the symbol server. Or, a pointer to the location of the symbol can be copied to the symbol server. If you have an archive folder that contains your old builds, you may want to index pointers to the PDB files that are already on the share, instead of duplicating symbols. Because symbols can sometimes be tens of megabytes in size, it's a good idea to plan ahead for how much space you may require to archive all the builds of your project throughout development. If you index only pointers to symbols, you may have problems if you remove old builds, or change the name of a file share.

For example, to index recursively all the symbols in c:\dxsym\Extras\Symbols that you obtained from the October 2006 DirectX SDK onto a symbol server file share called \\mainserver\symbols, you can use the following command:

"c:\Program Files\Debugging Tools for Windows\symstore" add /f "C:\dxsym\Extras\Symbols\*.pdb"
/s \\mainserver\symbols /t "October 2006 DirectX SDK " /r

The /t "comment" parameter is used to add a description to the transaction that added the symbols. This can be useful when performing administrative tasks on the symbols.

Best Practices

• Set up your own symbol server file share for your team, company, or product.
• Set up _NT_SYMBOL_PATH to point to a local cache, to a private symbol server, and to the Microsoft symbol server.
• If a debugger cannot load symbols for a component you are debugging, contact the owner of the component to request symbols—at least a stripped PDB.
• Set up an automated build system to index symbols on your private symbol server for each build that is produced. Make sure that the builds that you distribute are the builds that are generated by this process. This ensures that symbols are always available to debug problems.
• Set up a symbol server to allow debuggers to access the source code for a specific module directly from a Visual Source Safe or Perforce based source control system. If the source file information and symbols for a released version of a game are indexed, developers who have access to your symbol server can have full source level debugging of reported issues, without keeping build environments or old versions of source files on their development computers. To set up your symbol server to allow indexing of source file information, see source server documentation.