jasone's blog - random utterings from a Microsoft support guy

How to setup your PC to collect dumps for an application crash

Jason_E — Mon, 16 Jul 2018 14:23:58 GMT

There are lots of ways to do this, but this is my favorite basic setup.

Download the latest version of procdump from here
Create a folder on a drive that has enough disk space to store the application dumps, e.g. C:\Dumps
Extract the procdump application to this folder
Open an Administrative command prompt and navigate to your new folder
Run procdump /ma /i
Wait for the application(s) to crash as indicated by an Application Error Event ID 1000 in the application log
The dump(s) will be located in the target folder (from step 2, e.g. C:\Dumps)

Changing the image used in the "Force a specific default lock screen and logon image" policy

Jason_E — Thu, 21 Jun 2018 13:03:02 GMT

Customer called that was using the "Force a specific default lock screen and logon image" policy on Windows 8.1 to do exactly what the policy says. After setting this up, they wanted thought it would be cool to periodically change the image. They did not want to have to go through all the steps to change the policy, so they just wanted to overwrite the file that was being used in the policy with a different image. They tested doing this and found that the replacing the image did update the lock screen, so... they called support. The support engineer they were working with tested this and found that it did not work for them either. This is the point I was engaged with "Hey, there is a bug in Windows, this policy does not work". Let me start by saying I have never used this policy or troubleshot this policy, so I did not have any background of how it worked, so I took the engineers comments at face value. Here is my conversation with the support engineer:

Me: Does this work on Windows 10?
Other support guy: They are not using Windows 10
Me: Fair enough, can you test to see if works on Windows 10?
Other support guy: Why? They are not running Windows 10.
Me : Well.. if it works on Windows 10, we will know this issue was something we found and fixed. If it does not work on Windows 10, we will know this has always been this way, and we will make a decision on what to do next from there.
Other support guy: OK...

Later...

Other support guy: OK.. I tested it and it works on some version of Windows 10, and does not work at all on others.
Me: What do you mean?
Other support guy: I tested it on 1703 and it worked, but it does not work at all on 1709 or 1803 (see here for a quick decoder ring of these build numbers)
Me: What do you mean when you say it "does not work at all"?
Other support guy: The background image I am trying to use never gets applied to the lock screen.

My internal monologue: Why would this policy work on some version of Windows 10, and not all of them? Why would it work on an earlier version and be broken in later versions. I go back and read the policy help from top to bottom.

Me: Can you send me a screen shot of winver.exe from your 1709 and 1803 machines
Other support guy: Sure

I get email with screenshots...

Me: Your 1709 and 1803 machines are Windows 10 Professional.
Other support guy: So.
Me: Read the last line in the group policy description (for reference it reads: "Note: This setting only applies to Enterprise, Education, and Server SKUs")
Other support guy: Oh.

Later...

Other support guy: OK. It seems to work on all versions of Windows 10.
Me: OK, good to know.

Here is where we stand based of what we learned so far:

The policy does exactly what it says it will do in all OS that support the policy. The policy does not state that you should be able to change the image using the exact same name in the location where the file is stored and the OS will know it is different and use it. The policy says "If you want to pick A particular image for a lock screen, we can do that". From a technical perspective, my inner-nerd thinks what we are doing here should work, but the policy does not infer or imply that the image change part of this is going to work.
The act of replacing of the image with an image of the same name and having the OS automatically pickup the new image does not work on Windows 8.1, but does work on Windows 10.

Quick call with the customer and talked through our findings. Customer said they would do some testing on Windows 10 and report back. Customer follows up and says they cannot get this to work on any version of Windows 10. My internal monologue: Seriously... I did not do the testing, the other support guy did. Why do I trust anyone? :) Now I crank up my VM of Windows 10 and run through the test. Works like a champ. Email the customer and confirm it is working and explain the steps I am using. Customer comes back and states that these steps are not working for him. Hmm... back to my VM. I test it again, and this time mine does not work. What is going on here? I had created by backgrounds just by opening up paint and making an image, saving it to folder 1, then modifying it and saving a second image to folder 2, using the same name as the image in folder 1. Now that I am questioning everything, I deleted my previously created images and made new ones. I now have lockScreen.jpg images stored in two different folders, one is just a different color background. I run through the test taking the first image and putting it on the desktop then modifying the policy to use the image on the desktop and running gpupdate /force. I lock the desktop, and my image is on the lock screen. I then take the image out of folder 2 and replace the image on the desktop and then lock the desktop. The image on the lock screen changes to my second image. So far, so good. Just to be sure, let's do this again. I take the image out of folder 1 and replace the image on the desktop. I lock the desktop... and... still has the image from folder 2. Why did it work the first time, but not the second time? On a hunch, I right clicked on the image on the desktop, went to properties and over to the details tab and edited the "Comments" meta-data field. Just typed in "foo". Clicked OK and locked my desktop. Boom! The image switched to the updated lockScreen.jpg. This test makes me think that if the image is older (last modified date) than the one we currently have, the OS ignores it. In order to be sure I understand how this works, I wanted to find the code that updates the images. In order to do this I setup Sysinternal's Process Monitor (Procmon) to monitor for my image name. I kicked off a trace and locked my machine and unlocked it. I then looked at the trace for what process was interacting with my image. It was LogonUI.exe and from this I was able to get the callstack for the interaction with my image. Here is what I found: 0 FLTMGR.SYS FltpPerformPreCallbacks+0x2dc 1 FLTMGR.SYS FltpPassThroughInternal+0x8c 2 FLTMGR.SYS FltpCreate+0x2c9 3 ntoskrnl.exe IofCallDriver+0x59 4 ntoskrnl.exe IopParseDevice+0x773 5 ntoskrnl.exe ObpLookupObjectName+0x73b 6 ntoskrnl.exe ObOpenObjectByNameEx+0x1df 7 ntoskrnl.exe NtQueryFullAttributesFile+0x19e 8 ntoskrnl.exe KiSystemServiceCopyEnd+0x13 9 ntdll.dll ZwQueryFullAttributesFile+0x14 10 KernelBase.dll GetFileAttributesExW+0x95 11 Windows.UI.Immersive.dll CLockScreenHistory::_GPCacheStateFromEntry+0x49 12 Windows.UI.Immersive.dll CLockScreenHistory::_InitOrUpdateGPImages+0x80 13 Windows.UI.Immersive.dll CLockScreen::_GetImageForUser+0x112a 14 Windows.UI.Immersive.dll CLockScreen::GetImageRandomAccessStreamForUser+0x70 15 Windows.UI.Immersive.dll CLogonUILockScreenStore::GetImageRandomAccessStreamForUser+0x82 16 Windows.UI.Logon.dll ::operator()+0x13a 17 Windows.UI.Logon.dll std::_Func_impl 18 Windows.UI.Logon.dll std::_Func_class::_InitialTaskHandleCompareFileTime API to see if the modified date is earlier, same, or later than the one in the cache. If the image is newer (later in time) than the one in the cache, we update the file in the cache. This explains why my testing would work then fail. When I created my images I created the first one, then created the second one. When I tried to use them in that order, it worked. But when I tried to toggle back from the second image to the first image, the date modified for the first file is before then one in the cache, and the OS does not update the cache, leaving us with a lock screen that does not change. So... why didn't this work on Windows 8.1? Back to the source for Windows 8.1 and we find the source is the same. Boot a Windows 8.1 machine, run through a test with what we know now, and sure enough, it works there to. There are a lot of lessons to learn from this one:

Don't assume the problem presented in accurate (Trust, but verify).
Just because something does not work, does not mean it is a bug.
Just because it worked once, does not mean it will always work.
When you are not sure what data to collect, collect a Procmon. This is one of the best tools for reversing what is responsible for what and how these components work.

- Jason E

Windows 10 - RS4/1803 - Command prompt error: "Not enough memory resources are available to process this command"

Jason_E — Thu, 07 Jun 2018 13:37:16 GMT

Opening a command prompt (cmd.exe) when booted from the Windows 10 (1803) ADK WinPE, WinRE, or when booted from the Windows 10 1803 install media you will encounter the following error: "Not enough memory resources are available to process this command"

If you open a command prompt on your machine (not booted to one of the environments outlined above), you will see a copyright string that looks like:

During a refactoring of the components responsible for keeping these copyright strings, a resource file was accidentally not included. As a result, when the command prompt goes to read the string, it cannot find it, and the code assumes the reason the string was not found is due to a low memory condition.

What you should know:

There is no low memory condition. It is just a bad error message.
This issue has no impact on any functionality
This error can be safely ignored.
This error has been corrected in the next version of Windows 10 (RS5).
If you are seeing this error and you are not booted into a RS4 pre-install environment, this is not your issue.

If you encounter this error, and it caused anything, e.g. confusion, outrage, stopped a deployment, etc., please leave a comment. We are trying to determine if it makes sense to backport the RS5 fix to RS4. Thanks!

Public service announcement regarding new hotkey: CTRL+WIN+C = where did all the color go?

Jason_E — Fri, 16 Feb 2018 20:16:08 GMT

I am running along using my happy Windows 10 PC. I was trying to do something fancy keyboard hotkey and all of a sudden all my apps on all monitors went black and white (a.k.a. gray/grey grayscale/greyscale... kinda Ansel Adams photo looking). I am a keyboard guy. I really like hotkeys, but sometimes there is a misfire where my fingers don't exactly do what my brain told them to do. I look at my screens, one eyebrow raised. What just happened. Did my video card just crap out? Maybe a loose cable? But why would that cause both monitors... hmm. What have I done. Do I need to reboot? Before doing anything else I hit the Windows key and type "grayscale". Hmm... what is this? "Turn color filters on or off". Clicking this brings you to settings. Here you learn that we have added a hotkey, CTRL+WIN+C to turn on and off this option. This is not the key sequence I was attempting, but it was really close. High five to the person who made sure the grayscale would show this setting. Just thought this might help a fellow keyboard hotkey user.

I love hotkeys... WIN+ALT+D

Jason_E — Wed, 12 Jul 2017 14:46:11 GMT

I am frequently writing an email where I need to take a peek at a calendar to determine what the date is for some day next week. The new Windows 10 calendar that is under the click of the clock is great, but I have always wanted a way to use this calendar via a keyboard hotkey. Not sure which Windows 10 version added it, but my RS2/Creators Update version now supports WIN+ALT+D which opens up the calendar for me. Love it.

What is CDD.sys?

Jason_E — Wed, 12 Jul 2017 14:41:36 GMT

I frequently run across drivers running on Windows and need to figure out what the driver is for and who owns it. Today's driver is CDD.sys. It was being called into by Win32k.sys, the kernel side of the Windows Messaging subsystem on Windows, indicating something to do with graphics. I was able to determine this driver is the Canonical Display Driver, which is a practically meaningless name other than indicating something to do with graphics, but now I know it is part of the OS's graphics sub-system. Good to know.

What is an Access Violation?

Jason_E — Wed, 12 Nov 2014 06:42:33 GMT

An access violation (AV) occurs when the program performs and action on a memory address that does not align with the page protection for that memory.

3 common types:

Read
Write
Execute from non-executable

In order to determine the cause for the AV, you first need to know which type you are dealing with. With both the read and write scenario, you evaluate the address that is referenced and try to figure out why the address got to be in such a state that would cause an AV.

Address Zero

The most common AV pattern will be a read or write to address 0 (zero). A common programming issue involves several scenarios where the programmer expects a value to be a valid address, but for whatever reason it did not get set properly (e.g. a function failed and returned 0 instead of the expected value). In order to address this common mistake, the OS sets the first page of memory in each process up as a trap (think big metal thing to catch an animal) to catch this issue. The trap is created by marking the first page of memory in each process, address 0, to be NO_ACCESS.

Reference: Reserved Virtual Address Ranges

0:001> !address 0

Usage:                  Free
Base Address:           00000000
End Address:            00010000
Region Size:            00010000
State:                  00010000 MEM_FREE
Protect:                00000001 PAGE_NOACCESS <--- We can see the page protection on this address is NO ACCESS

Type:

Execute from non-executable (DEP - Data Execution Prevention)

Another AV can occur if the instruction pointer is ever set to memory that does not include the option to enable execution.

Reference: Data Execution Prevention (Wikipedia)

If we look at the exception record, we see it was a DEP access violation

0:034> .exr -1
ExceptionAddress: 00610524
   ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000008
   Parameter[1]: 00610524
Attempt to execute non-executable address 00610524

In this case, we don't look at the failing instruction, we focus on the fact that EIP was set to a value that did not have the execute flag.

0:034> r
eax=00000000 ebx=00610000 ecx=0182ffb4 edx=7c82ed54 esi=00000000 edi=00000000
eip=00610524 esp=0182ffbc ebp=0182ffec iopl=0         nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000246
00610524 55              push    ebp

0:034> kvL
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0182ffb8 77e6608b 00610000 00000000 00000000 0x610524
0182ffec 00000000 00610524 00610000 00000000 kernel32!BaseThreadStart+0x34

0:034> !address 00610524

Usage:
Base Address:           00610000
End Address:            00611000
Region Size:            00001000
State:                  00001000 MEM_COMMIT
Protect:                00000004 PAGE_READWRITE <-- does not include execute
Type:                   00020000 MEM_PRIVATE
Allocation Base:        00610000
Allocation Protect:     00000004 PAGE_READWRITE

Compare this page protect to that of a Windows function

0:034> !x kernel32!ReportEventW
77ec0285 kernel32!ReportEventW

0:034> !address 77ec0285

Usage:                  Image
Base Address:           77e41000
End Address:            77ecb000
Region Size:            0008a000
State:                  00001000 MEM_COMMIT
Protect:                00000020 PAGE_EXECUTE_READ <-- does include execute
Type:                   01000000 MEM_IMAGE
Allocation Base:        77e40000
Allocation Protect:     00000080 PAGE_EXECUTE_WRITECOPY
Image Path:             C:\WINDOWS\system32\kernel32.dll
Module Name:            kernel32
Loaded Image Name:      kernel32.dll

What is a deadlock?

Jason_E — Wed, 12 Nov 2014 06:39:26 GMT

A deadlock is a situation in which two or more competing actions are each waiting for the other to finish, and thus neither ever does. Avoiding deadlocks is all about making sure any locks that are acquired in a series (A, B, C, etc.) are always acquired in the same order. For example, say we have locks A and B. Thread 1 always acquires these locks A first, then B. Thread 2 always acquires them B first, then A. If at the exact same time, both of these threads run and thread 1 is at the point where it has grabbed lock A, but not yet grabbed lock B, and at this point thread 2 has grabbed lock B and is ready to grab lock A, we have reached a condition where both threads are stuck forever. This is a deadlock. The chances of this occurring are impacted by how often the locks are acquired in different orders AND by the time that passes between the acquisition of lock A and B. The more time that passes between the point lock A is acquired and the point lock B is acquired, the more likely thread 2 is to run and cause our issue.

Example:

The following thread (thread 1) was attempting to apply group policy. While do this operation it called into shell32 to get a known folder path. Shell32 needed to load a dll to complete the operation. To accomplish the load, this thread needed to acquire the LdrpLoaderLock (this is lock 2). Although you cannot see this yet, this thread has already acquired a different Critical Section (lock 1) in the Shell32!kfapi class.

0: kd> !mex.t fffffa80132087b0
Process                                     Thread                 CID       TEB              UserTime KernelTime ContextSwitches Wait Reason     WaitTime State
svchost.exe (GPSvcGroup) (fffffa800fb73040) fffffa80132087b0 (E/K) c360.c694 000007fffff92000     .016       .078             987 UserRequest 13:57:51.562 Waiting

WaitBlockList:
    Object           Type                 Other Waiters
    fffffa8011df5040 SynchronizationEvent            40

# Child-SP         Return           Call Site                                                     Info
0 fffffa6025f24980 fffff80001857dfa nt!KiSwapContext+0x7f
1 fffffa6025f24ac0 fffff8000184ca0b nt!KiSwapThread+0x13a
2 fffffa6025f24b30 fffff80001ac5428 nt!KeWaitForSingleObject+0x2cb
3 fffffa6025f24bc0 fffff800018555b3 nt!NtWaitForSingleObject+0x98
4 fffffa6025f24c20 0000000077166b5a nt!KiSystemServiceCopyEnd+0x13
5 000000000249c928 00000000771454aa ntdll!ZwWaitForSingleObject+0xa
6 000000000249c930 00000000771453a1 ntdll!RtlpWaitOnCriticalSection+0xea
7 000000000249c9e0 000000007716d637 ntdll!RtlEnterCriticalSection+0xf4                            Critical Section: ntdll!LdrpLoaderLock Owning Thread: e2d8
8 000000000249ca10 00000000771539c9 ntdll!LdrLockLoaderLock+0x137
9 000000000249ca50 0000000076f3bfc0 ntdll!LdrLoadDll+0xf9
a 000000000249cd40 0000000076f48c26 kernel32!LoadLibraryExW+0x3a2
b 000000000249cdd0 000007fefd970b71 kernel32!LoadLibraryA+0x46
c 000000000249ce00 000007fefd970af7 SHELL32!__delayLoadHelper2+0x85
d 000000000249ce90 000007fefd961969 SHELL32!_tailMerge_ole32_dll+0x3f
e 000000000249cf00 000007fefd961140 SHELL32!kfapi::CRegistryKeyProvider::OpenDefinitionKey+0x61
f 000000000249cfb0 000007fefd9617f6 SHELL32!kfapi::CFolderDefinitionStorage::LoadRegistry+0x92
10 000000000249d1c0 000007fefd96159d SHELL32!kfapi::CFolderDefinitionStorage::Load+0x62
11 000000000249d3d0 000007fefd9606cc SHELL32!kfapi::CFolderDefinitionCache::Load+0x111
12 000000000249d5a0 000007fefd97bff0 SHELL32!kfapi::CFolderPathResolver::GetPath+0xb8
13 000000000249d940 000007fefd97c492 SHELL32!kfapi::CFolderCache::GetPath+0x153
14 000000000249da40 000007fefd97c3b6 SHELL32!kfapi::CKFFacade::GetFolderPath+0x9a
15 000000000249daf0 000007fefd91d7bd SHELL32!SHGetKnownFolderPath_Internal+0x8c
16 000000000249db60 000007fef2469b70 SHELL32!SHGetKnownFolderPath+0x1c
17 000000000249db90 000007fef246796b fdeploy!CFileCacher::Init+0x70
18 000000000249dc10 000007fef246333f fdeploy!CPolicyComputant::GetRedirectionInfo+0x1a7
19 000000000249e400 000007fef2465571 fdeploy!CEngine::ProcessGroupPolicyEx+0x20b
1a 000000000249e4c0 000007fefbaf1e73 fdeploy!ProcessGroupPolicyEx+0x1f9
1b 000000000249e570 000007fefbaf0088 gpsvc!ProcessGPOList+0x637
1c 000000000249e8f0 000007fefbaebfd5 gpsvc!ProcessGPOs+0x2c50
1d 000000000249f720 000007fefbb381ad gpsvc!ApplyGroupPolicy+0x7d5
1e 000000000249f9c0 000007fefbb3b645 gpsvc!CDefaultPolicyApplier::ApplyGroupPolicy+0x4d
1f 000000000249fa10 000007fefbb3b124 gpsvc!CGroupPolicySession::ApplyGroupPolicyForPrincipal+0x4e1
20 000000000249fae0 0000000076f3aefd gpsvc!CGroupPolicySession::ApplyGroupPolicyThread+0x30
21 000000000249fb20 0000000077146591 kernel32!BaseThreadInitThunk+0xd
22 000000000249fb50 0000000000000000 ntdll!RtlUserThreadStart+0x1d

In order for thread 1 (above) to move forward, we need to find what the current owner of the loader lock (lock 2) is doing. We see the thread below is in the loader (look for ntddll!LdrLoadDll which leads to gpprefcl!dllmain. So this thread (thread 2) acquired the loader lock (lock 2), but is waiting on a critical section (the one for shell32!kfapi) owned by the thread above (thread 1)

0: kd> !mex.t -t e2d8
Process                                     Thread                 CID       TEB              UserTime KernelTime ContextSwitches Wait Reason     WaitTime State
svchost.exe (GPSvcGroup) (fffffa800fb73040) fffffa8012193bb0 (E/K) c360.e2d8 000007fffffd8000     .016       .047             717 UserRequest 13:57:50.921 Waiting

WaitBlockList:
    Object           Type                 Other Waiters
    fffffa801063b370 SynchronizationEvent             3

# Child-SP         Return           Call Site                                                     Info
0 fffffa602529e980 fffff80001857dfa nt!KiSwapContext+0x7f
1 fffffa602529eac0 fffff8000184ca0b nt!KiSwapThread+0x13a
2 fffffa602529eb30 fffff80001ac5428 nt!KeWaitForSingleObject+0x2cb
3 fffffa602529ebc0 fffff800018555b3 nt!NtWaitForSingleObject+0x98
4 fffffa602529ec20 0000000077166b5a nt!KiSystemServiceCopyEnd+0x13
5 000000000188d2f8 00000000771454aa ntdll!ZwWaitForSingleObject+0xa
6 000000000188d300 00000000771453a1 ntdll!RtlpWaitOnCriticalSection+0xea
7 000000000188d3b0 000007fefd97bb53 ntdll!RtlEnterCriticalSection+0xf4                            Critical Section: 0000000003fce220 Owning Thread: c694
8 000000000188d3e0 000007fefd9606cc SHELL32!kfapi::CFolderDefinitionCache::Load+0x4b
9 000000000188d5b0 000007fefd96206d SHELL32!kfapi::CFolderPathResolver::GetPath+0xb8
a 000000000188d950 000007fefd97c492 SHELL32!kfapi::CFolderCache::GetPath+0x33b
b 000000000188da50 000007fefd97c3b6 SHELL32!kfapi::CKFFacade::GetFolderPath+0x9a
c 000000000188db00 000007fefd97d03c SHELL32!SHGetKnownFolderPath_Internal+0x8c
d 000000000188db70 000007fefd9623d9 SHELL32!SHGetFolderPathEx+0x32
e 000000000188dbc0 000007feee4ae19e SHELL32!SHGetFolderPathW+0xed
f 000000000188dc30 000007feee4623f2 gpprefcl!apmSHGetFolderPath+0x7a
10 000000000188dc70 000007feee462843 gpprefcl!apmClientTraceBase::initialize+0x102
11 000000000188dda0 000007feee462215 gpprefcl!apmClientTraceBase::TraceFormat+0x27
12 000000000188ddd0 000007feee4857b1 gpprefcl!apmTraceFormat+0x49
13 000000000188de90 000007feee49753f gpprefcl!DllMain+0x6d
14 000000000188dec0 000000007715422d gpprefcl!__DllMainCRTStartup+0xbf
15 000000000188e020 0000000077161a28 ntdll!LdrpRunInitializeRoutines+0x1f6
16 000000000188e200 0000000077153a06 ntdll!LdrpLoadDll+0x4b1
17 000000000188e520 0000000076f3bfc0 ntdll!LdrLoadDll+0x136
18 000000000188e810 000007fefbaf165c kernel32!LoadLibraryExW+0x3a2
19 000000000188e8a0 000007fefbaf1c66 gpsvc!LoadGPExtension+0x40
1a 000000000188e8d0 000007fefbaf0088 gpsvc!ProcessGPOList+0x42a
1b 000000000188ec50 000007fefbaebfd5 gpsvc!ProcessGPOs+0x2c50
1c 000000000188fa80 000007fefbb381ad gpsvc!ApplyGroupPolicy+0x7d5
1d 000000000188fd20 000007fefbb3b645 gpsvc!CDefaultPolicyApplier::ApplyGroupPolicy+0x4d
1e 000000000188fd70 000007fefbb3b124 gpsvc!CGroupPolicySession::ApplyGroupPolicyForPrincipal+0x4e1
1f 000000000188fe40 0000000076f3aefd gpsvc!CGroupPolicySession::ApplyGroupPolicyThread+0x30
20 000000000188fe80 0000000077146591 kernel32!BaseThreadInitThunk+0xd
21 000000000188feb0 0000000000000000 ntdll!RtlUserThreadStart+0x1d

Note that this condition would not have occurred if these two thread did not run at the same time. Now... How do you fix this. In this particular case the module gpprefcl.dll is breaking the rules for what is allowed inside the dllmain funciton. If you refer to MSDN on the details for dllmain, you will see it clearly says keep it short and simple. The issue above resulted in a hotfix that removed the logging code from the dllmain function. This negated the existing dependency on shell, which avoided the issue all together.

Not all deadlocks involve the loader lock and dllmain. In other cases the rule is "make sure locks are acquired in the same order every time". Lock 1, then Lock 2. If you have code that aquires the same locks in reverse while another thread tries in the normal direction, you have the potential to deadlock. The time between the acquisition of lock 1 and lock 2 combined with how frequently the locks are acquired will determine how likely a deadlock is going to occur.

Started means started, not completely ready.

Jason_E — Wed, 27 Aug 2014 14:29:14 GMT

Windows is extensible. Extensible means we have places that are built for external software writers to plug in to provide additional functionality that was not included by default. This makes Windows very powerful as a platform. This comes at a price. In any plug-in model, you have the option to document the contract or to document the contract and ENFORCE it. For the most part, Windows does not enforce these contracts.

Let’s walk through an example. On Windows we have the concept of services. Services are apps that run in the background, automatically even if the user is not logged in. These services can have a few different startup options, but a common one is Automatic. This means the service will be started early during the boot of the system. Each of these services is asked to start, and as soon as they are started, they are supposed to report back started. Here is the contract for a service and how long they have to go from “Asked to start” to “Started” @ Writing a Servicemain Function (MSDN):

Excerpt from this page:

The sample initialization function, SvcInit, is a very simple example; it does not perform more complex initialization tasks such as creating additional threads. It creates an event that the service control handler can signal to indicate that the service should stop, then calls ReportSvcStatus to indicate that the service has entered the SERVICE_RUNNING state. At this point, the service has completed its initialization and is ready to accept controls. For best system performance, your application should enter the running state within 25-100 milliseconds

I have highlighted the key line. The contract, despite being delivered softly, is “Dear service, you have 25 milliseconds to get to the started state”. The problem is the service writer’s perception of what “Started” means. Our intent, was “Started” means you have crossed the start line. Many developers believe this means their service has reached a point where they feel they are FULLY operational. Why does this matter? Well, automatic services are started in a serialized manner. First the Service Control Manager (SCM) reads the list of services out. It first orders the list based on each services LoadOrderGroup,then each group is re-ordered them based on dependencies. To see the outcome of this sorting, you can use LoadOrder from Sysinternals. So we have a list of services, divided in to groups, that are reshuffled to take care of dependencies. The list ends up being, for the most part, alphabetically sorted based on the service name.

If a service that is marked as automatic decides that he is going to take a long time to start, but while doing this, continuously reports pending back to service control manager, then all the services that have not started yet, have to wait. In addition, any request by anything already running to start a service is blocked too. So a service named “AAA Service” that is marked as automatic, begins is startup, and decides to do a lot of work, or connect to something over the network, or any other bad decision, while still reporting back to SCM pending has blocked the box from allowing you to have a quick boot.

I have seen this as a frequent cause of hangs during boot. Here is a diagram from a case I worked where a customer hit this issue:

The bad behavior is the contract violation of the aaaService. As the end user, you are stuck staring at a logon screen waiting to get to your desktop because the application that collects your credentials, Winlogon, called a function that required the NLA service to start. Since SCM is blocked waiting on the “bad” service to report started, Winlogon’s request is blocked.

Our team is hiring. We debug Windows

Jason_E — Wed, 27 Aug 2014 10:37:25 GMT

I work on a team that debugs windows. Pre-snarky “Yes, I stay busy”. This means I work in Microsoft Support, talking to customers and internal support staff about two key types of issues:

Crash, hang, or dump.
Configured correctly and still not working.

The Tools:

Debugging Tools for Windows (Windbg)
Windows Performance Toolkit (XPerf)
Anything else we need to solve an issue:

All tools that end with “mon” (i.e. Procmon, Netmon, etc.)

The skills

Above all others: Ability to work with people: customers, internal staff, internal teams, etc.
Culture fit: When we interview, we look for people with values that align well with the Zappos Core Values
Troubleshooting: Think Kepner Tregoe’s customer issue resolution
Software critic (ability to read and understand code): We read a lot of Windows code. This includes assembly, C, C++, C#, STL, ATL, blah blah. Lots of different code, with a wide variety of programmers.
Crazy fast learning skills. We, Microsoft are shipping new versions at a rapid pace. These new versions come with new features. When they don’t work, we have to go from 0-60 in a very shot amount of time.
Great teachers, but better students. We need people to make us better at what we do. This involves sharing what you know. At the same time, we need to have people that are willing to be great students. Great listeners. Great note takers. People who are learning in order to teach. If you can’t teach it, you don’t know it.

We take a problem, determine what data is needed, and use this data to determine what the problem is. If the problem is with Microsoft code, we engage the product group and work with them to get a fix. Net result is a hotfix/Windows Update.

This is the hardest job I have ever had. It is challenging, but always fun. There is a lot of satisfaction is being able to take a reported problem, and follow that through to a fix that will touch every Windows machine in the world.

Job posting here