Useful service tricks - Debugging service startup
For the better part of the past 15 years, I've been working on one or another services for the Windows platform (not always services for windows, but always services ON windows).
Over that time, I've developed a bag of tricks for working with services, I mentioned one of them here. Here's another.
One of the most annoying things to have to debug is a problem that occurs during service startup. The problem is that you can't attach a debugger to the service until it's started, but if the service is failing during startup, that's hard.
It's possible to put a Sleep(10000) to cause your service startup to delay for 10 seconds (which gives you time to attach the debugger during start), that usually works, but sometimes service startup failures only happen on boot (for autostart services).
First off, before you start, you need to have a kernel debugger attached to your computer, and you need the debugging tools for windows (this gets you the command line debuggers). I'm going to assume the debuggers are installed into "C:\Debuggers", obviously you need to adjust this for your local machine.
One thing to keep in mind: As far as I know, you need have the kernel debugger hooked up to debug service startup issues (you might be able to use ntsd.exe hooked up for remote debugging but I'm not sure if that will work).
This of course begs the next question: "The kernel debugger? Why on earth do I need a kernel debugger when I'm debugging user mode code?". You're completely right. But in this case, you're not actually using the kernel debugger. Instead, you're running using a user mode debugger (ntsd.exe in my examples) that's running over the serial port using facilities that are enabled by the kernel debugger. It's not quite the same thing.
There are multiple reasons for using a debugger that's redirected to a kernel debugger. First off, if your service is an autostart service, it's highly likely that it starts long before the a user logs on. So an interactive debugger won't really be able to debug the application. Secondly, services by default can't interact with the desktop (heck, they often run in a different TS session from the user (this is especially true in Vista, but it's also true on XP with Fast User Switching), so they CAN'T interact with the desktop). That means that when the debugger attempts to interact with the user, it can't because it flat-out can't because the desktop is sitting in a different TS session.
There are a couple of variants of this trick, all of which should work.
Lets start with the simplest:
If your service runs with a specific binary name, you can use the Image File Execution Options registry key (documented here) to launch your executable under the debugger. The article linked shows how to launch using Visual Studio, for a service, you want to use the kernel debugger, so instead of using "devenv /debugexe" for the value, use "C:\Debuggers\NTSD.EXE -D", that will redirect the output to the kernel debugger.
Now for a somewhat more complicated version - You can ask the service controller to launch the debugger for you. This is useful if your service is a shared service, or if it lives in an executable that's used for other purposes (if you use a specific -service command line switch to launch your exe as a service, for example).
This one's almost easier than the first.
From the command line, simply type:
sc config <your service short name> binpath= "c:\debuggers\ntsd.exe -d <path to your service executable> <your service executable options>
Now restart your service and it should pick up the change.
I suspect it's possible to use the ntsd.exe as a host process for remote debugging, I've never done that (I prefer assembly language debugging when I'm using the kernel debugger), so I don't feel comfortable describing how to set it up :(
Edit: Answered Purplet's question in the comments (answered it in the post because it was something important that I left out of the article).
Edit2: Thanks Ryan. s/audiosrv/<your service>/
Comments
Anonymous
March 01, 2006
I've written a service one time only (and it could run also as a console application so most of the debug was done in console mode) but I remember I just debugged it using devenv.
Why is the kernel debugger needed for a standard service (which runs in user mode) ?Anonymous
March 01, 2006
The comment has been removedAnonymous
March 01, 2006
I may have missed something, but I put __asm int 3 in my code when I the ability to have a debugger come up and windows puts up that nice dialog asking if I want to attach a debugger to the process.
Am I missing a big point here?Anonymous
March 01, 2006
I may have missed something, but I put "__asm int 3;" in my code where I want to debug and windows puts up that nice dialog asking if I want to attach a debugger to the process.
Am I missing a big point here?Anonymous
March 01, 2006
>sc config audiosrv binpath= "c:debuggersntsd.exe -d <path to your service executable> <your service executable options>
Presumably, change "audiosrv" to the name of your service as well.Anonymous
March 01, 2006
Rob,
The __asm int 3 (or calling DebugBreak() which works cross platform) relies on the fact that the AE debugger will catch and launch. First off, it doesn't work at system startup (there's no user to catch the debugger). Second, I'll be honest and say that I don't trust the AEDebug stuff, it's not reliable enough for my tastes (especially on Vista where session 0 isn't the first logon session).Anonymous
March 01, 2006
Another thing that should be mentioned is that if the service does not respond to the SCM's start request within 30 seconds, the SCM will terminate the service process. If you're poking around in the debugger (or waiting to attach the debugger) that 30 seconds can fly by and suddenly your service is gone.
The timeout can be increased using the registry value:
HKLMSYSTEMCurrentControlSetControlServicesPipeTimeout
Set it to the number of milliseconds (default is 30000 = 30 seconds).Anonymous
March 01, 2006
The comment has been removedAnonymous
March 01, 2006
Is "TS session" the new name for WindowStation?Anonymous
March 01, 2006
> Secondly, services by default can't interact with the
> desktop (heck, they often run in a different TS session from
> the user (this is especially true in Vista, but it's also true on
> XP with Fast User Switching), so they CAN'T interact with
> the desktop).
That was my understanding too, and it posed a big problem, so I posted in one of the Microsoft public newsgroups (probably the kernel group) and Microsoft gave an answer I wasn't expecting. Microsoft said that a service can still open Window Station 0 and still get the desktop of whichever user is logged in at the console.
Of course we don't want a service to pump messages from the user's console, but that's not an issue here. The service calls the SendInput() API under conditions where the user has asked the service to do so. We want SendInput's target to be in the desktop of that user, not some unsuspecting other user. I sure hope Microsoft's answer is going to remain correct.
Wednesday, March 01, 2006 5:10 PM by mikeb
> Another thing that should be mentioned is that if the service
> does not respond to the SCM's start request within 30
> seconds, the SCM will terminate the service process.
Quiz: Under what condition will the SCM terminate the service process a lot quicker than 30 seconds, and record a message in the event log saying that the service didn't respond for 30 seconds even when you've been watching and 30 seconds haven't passed yet?
This happened to me enough times that I've learned to recognize it and remember the answer_;;Anonymous
March 01, 2006
Why tunnel through KD? You can do it through serial from usermode:
ntsd -server com:port=COMPort,baud=BaudRate
Or better yet, TCP/IP:
ntsd -server tcp:port=1234
I guess this won't work when you're debugging something that prevents networking/COM from working, but how often does that happen?Anonymous
March 02, 2006
Jonathan, I did say it might work, I've never been able to make it work though, so I didn't include it.
chiph: No, you can have multiple windowstations per TS session.Anonymous
March 02, 2006
Controlling ntsd through kd is something that has been around for a long time - longer than the other methods of remote debugger control (i.e. process servers). Since the start, this has been the way you always debugged things like winlogon or (in particular) CSRSS.
CSRSS debugging is where this feature originated from, I suspect, as when you are doing CSRSS debugging the debugger is unable to call any functions that are ultimately sent to the CSRSS LPC port for further processing (which includes many functions in kernel32 (including almost all console I/O functions) or user32/gdi32 (especially in NT 3.x - almost all GUI output in that case). Because of these limitations, your allowed I/O functions are restricted to a very minimal subset - the kernel debugger connection being one of the main ways to interact with the user without going through CSRSS.
In more modern Windows versions, these restrictions are not quite as severe, although there are still a number of functions you can't call while CSRSS is frozen in the debugger or you would deadlock.Anonymous
March 02, 2006
The comment has been removedAnonymous
March 02, 2006
mithuns, absolutely. That's why I called out the sc config option, especially if it's coupled with the other linked trick.
Btw, svchost is internal-to-windows ONLY, so non Microsoft people, please ignore :)Anonymous
March 02, 2006
The comment has been removedAnonymous
March 03, 2006
The simple way around that is the age-old "at (current time plus one second) /interactive cmd.exe" which gives you a LocalSystem command prompt if you are logged on session zero.
Sadly, this trick will be going away with Vista - it's served me well, though.Anonymous
March 16, 2006
Process life support guidance.Anonymous
April 25, 2007
Just a few days ago, we had a discussion about ways to attach debuggers to processes whenever a process...Anonymous
July 10, 2007
Just a few days ago, we had a discussion about ways to attach debuggers to processes whenever a processAnonymous
June 08, 2009
PingBack from http://insomniacuresite.info/story.php?id=6321Anonymous
June 08, 2009
PingBack from http://cellulitecreamsite.info/story.php?id=1447