This article describes how to establish a live WinDbg connection to Linux. Live remote process debugging on Linux requires WinDbg version 1.2402.24001.0 or above.
The examples here use the Windows Subsystem for Linux (WSL), but other Linux implementations can also be used.
WinDbg types of remote process debugging
There are two primary methods of performing remote debugging with WinDbg - A process server or a KD connection server. Process servers are used for user-mode debugging; KD connection servers are used for kernel-mode debugging. See Process Servers (User Mode) and KD Connection Servers (Kernel Mode) for general information about these WinDbg connection types.
There are two ways to start debugging Linux user mode processes. You can either start gdbserver on a particular process or you can start gdbserver as a process server which can list and attach to existing processes. This is much like the DbgSrv (dbgsrv.exe) process server on Windows. For more information, see Activating a Process Server.
User mode Linux process debugging
It is possible to connect to a specific single user mode process, or in multi-mode, to see all of the process in a list and select one to connect to. Both methods are described in this topic. Both methods share the same connection string syntax, described next.
Format for gdbserver connection string
The format used to connect to gdbserver is "protocol:arguments" where arguments is a comma separated list of "argument=value". For a user mode gdbserver connection, the protocol is gdb and the set of arguments is as follows.
server=<address> - Mandatory: indicates the IP address of the gdbserver to connect to.
port=<port> - Mandatory: indicates the port number of the gdbserver to connect to.
threadEvents=<true|false> - Optional: indicates whether or not thread events for this version of gdbserver work correctly in stop mode.
There is an issue in the current gdbserver releases, where enabling thread events with a server in stop mode (which WinDbg uses) will cause gdbserver to crash. If this value is false (the default value), thread start and stop events will be synthesized but may appear significantly later than the actual time of thread creation/destruction. When a fix for this is available in gdbserver, the actual events can be enabled via this option.
Connecting to a Single User Mode Process
This section describes how to identify and connect to a single user mode process in Linux, using WinDbg.
WSL (Windows Subsystem for Linux)
The examples here use WSL (Windows Subsystem for Linux), but other Linux implementations can be used. For information about setting up and using WSL, see:
In this example walkthrough, we will connect to python3.
Locate the target system IP address
If connecting to a remote linux target, use a command such as ip route showto determine the external IP address.
linux
user1@USER1:/mnt/c/Users/USER1$ ip route show
default via 192.168.1.1 dev enp3s0 proto dhcp metric 100
172.25.144.0/24 dev enp3s0 proto kernel scope link src 192.168.1.107 metric 100
In this walkthrough, we will connect to WSL running on the same PC, and will use the IP address of localhost.
Attach GDBServer to the selected process
On the WSL linux console, enter gdbserver localhost:1234 python3 to start the gdbserver on port 1234, and attach it to the python3 process.
linux
USER1@USER1:/mnt/c/Users/USER1$ gdbserver localhost:1234 python3
Process python3 created; pid = 1211
Listening on port 1234
For some linux environments the command may need to be run as an administrator, for example using sudo - sudo gdbserver localhost:1234 python3. Use caution with enabling debugger administrator root level access and only use this when it is required.
Create the process server connection in WinDbg
Open WinDbg, and select "File / Connect to Remote Debugger" and enter a protocol string for the connection. For this example, we will use: gdb:server=localhost,port=1234.
Once you click the OK button, the debugger should connect to the gdbserver and you should be at the initial process start break.
Once you are at the initial breakpoint, you can hit 'g' several times. You will get module load messages (and sxe style "break on module load" events should work properly).
Note that it may take a moment to get to that point, as debug symbols are loaded into cache. In addition to looking for symbols and binaries via the symbol server or your local search path, the GDBServer integration has the capability of pulling these files from the remote filesystem if they cannot be found via symsrv or locally. This is typically a much slower operation than getting symbols from symsrv or a local search path but it makes the overall experience a better one by locating appropriate symbols.
Use the k stacks command to list the stack. It shows python3 modules, so this confirms that we are debugging python3 on Linux using WinDbg.
At this point, you should be able to do nearly everything that can be done using WinDbg attached to a remote Windows debugger over a remote process server. You can step, source level debug, set breakpoints, inspect locals, etc..
Once you are done debugging, use CTRL+D to exit the gbdserver window in WSL.
Connecting to a Process Server
In addition to connecting to a single process via a user mode GDBServer, you can set up one as a process server and list and attach to existing processes on the system. To do this, gdbserver is started with the "--multi" command line argument - gdbserver --multi localhost:1234
dbgcmd
user1@USER1:/mnt/c/Users/USER1$ sudo gdbserver --multi localhost:1234
Listening on port 1234
To connect to the process server, select "File / Connect to process server" in WinDbg and enter the same protocol string as you did with the single process gdbserver example above:
gdb:server=localhost,port=1234
Once you click the "OK" button, you should be connected to the gdbserver as a process server. As with dbgsrv, you can either spawn a new process or you can list the existing processes and attach to one.
In this example, use the "Attach to process" option.
Note that you will see many of the same things that are visible for Windows processes (including the PID, user, and command line). Some of the columns in the "attach to process" dialog are not relevant to Linux and will not contain data.
Ending the session
Use CTRL+D to exit the gbdserver window in WSL and select stop debugging in WinDbg. To end the session, in some cases you may need to exit the debugger.
Reconnecting to process server
WinDbg recognizes a "process server" versus a "single target" via whether the gdbserver is attached to a process or not. If you attach to some process, leave it frozen, close the debugger, and try to reconnect to the process server, in all likelihood, we will not recognize it as a process server. In this situation, restart the target gdbserver and reconnect the debugger.
Linux WinDbg functionality
While much of the functionality of the debugger will work as expected" in debugging core dumps (e.g.: stack walking, symbols, type information, local variables, disassembly, etc...), it is important to note that the entire debugging toolchain has not yet been made aware of ELF, DWARF, and the resulting differences from Windows semantics. Some commands in the debugger may currently result in unexpected output. For example, lm will still display incorrect information for an ELF module as it expects and manually parses PE headers.
Linux Kernel Mode via EXDI
The Windows debugger supports kernel debugging using EXDI. This allows debugging a wide variety of hardware and operating systems. For general information on setting up configuring and troubleshooting EXDI connections, see Configuring the EXDI Debugger Transport.
The DebugInfoD* tag can point to one or more DebugInfoD servers with each server URL formatted as https://domain.com and separated by *. The servers will be searched in the same order as listed in the source path and the files will be retrieved from the first matching URL. For more information, see Source Code Extended Access.
For more information on working with DWARF symbols and the Linux symbol utilities, such as !sourcemap and !diesym, see Linux symbols and sources.
C++ app walkthrough
Use a text editor (like nano or vi) to create your C++ file. For example:
nano DisplayGreeting.cpp
In the text editor, write your C++ program. Here’s a simple program that displays greetings, that needs to be debugged:
C++
#include<array>#include<cwchar>#include<cstdio>#include<iostream>usingnamespacestd;
voidGetCppConGreeting(wchar_t* buffer, size_t size){
wchar_tconst* const message = L"HELLO FROM THE WINDBG TEAM. GOOD LUCK IN ALL OF YOUR TIME TRAVEL DEBUGGING!";
wcsncpy(buffer, message, size);
}
intmain(){
std::array<wchar_t, 50> greeting{};
GetCppConGreeting(greeting.data(), greeting.size());
cin.get();
wprintf(L"%ls\n", greeting.data());
return0;
}
Save (CTRL-O) and exit (CTRL-X) the nano editor.
Compile the C++ file using g++. The -o option is used to specify the output file name, and the -g option generates a symbol file:
g++ DisplayGreeting.cpp -g -o DisplayGreeting
If there are no errors in your code, the g++ command will create an executable file named DisplayGreeting in your directory.
You can run the program using the following command:
./DisplayGreeting
Pressing the return key displays the message in the app. Looking at the output, it looks like the Greeting is being truncated, and "????" is being displayed instead.
HELLO FROM THE WINDBG TEAM. GOOD LUCK IN ALL OF YO????
Debugging DisplayGreeting
Once the code is ready to run, we can start the app using the gdbserver.
gdbserver localhost:1234 DisplayGreeting
Open WinDbg, and select "File / Connect to Remote Debugger" and enter a protocol string for the connection. For this example, we will use: gdb:server=localhost,port=1234.
Once connected the output should indicate that it is listening on port 1234 and that remote debugging connection is established.
dbgcmd
Bob@Bob6:/mnt/c/Users/bob$ gdbserver localhost:1234 DisplayGreeting
Process /mnt/c/Users/bob/DisplayGreeting created; pid = 725
Listening on port 1234
Remote debugging from host 127.0.0.1, port 47700
As mentioned previously, for some Linux environments the command may need to be run as an administrator, typically using sudo. Use caution with enabling debugger administrator root level access and only use this when it is required.
Add the source and symbol paths to the debugger session
To set breakpoints and view the source code and variables, set the symbols and source path.
For general information on setting the symbols path, see Using Symbols.
Use .sympath to add the symbol path to the debugger session. In this example, the code is running in this location in the WSL Linux Ubuntu, for a user named Bob.
\\wsl$\Ubuntu\mnt\c\Users\Bob\
In WSL this directory maps to the Windows OS location of: C:\Users\Bob\
Also supported is the automatic download of sources from DebugInfoD servers which support returning that artifact type. To take advantage of this, add the elfutils server using .srcpath.
Look through the code and note that the 50, may not be a sufficient size for the greeting message.
wchar_t const* const message = L"HELLO FROM THE WINDBG TEAM. GOOD LUCK IN ALL OF YOUR TIME TRAVEL
Confirm this by expanding the locals variable for the greeting and seeing that the greeting is truncated.
Troubleshooting the gdbserver connection
Use the --debug option to display additional information on the gdbserver console to gather more information on the connection status. For example to start a process server use this command.
Learn how to efficiently debug your .NET app by using Visual Studio Code to fix your bugs quickly. Use the interactive debugger within Visual Studio Code to analyze and fix your C# applications.