Win2008R2/Win7: STOP 0xF4 during Task Sequence / OS Deployment

Status: Resolved

Update 150630: one of my colleagues mailed me additional info, see below under Additional.

Recently I came across two very similar STOP 0xF4 cases, sent to me by ConfigMgr colleagues. When examining the dump, it became clear there was a problem with services.exe. In one dump it was still present while exiting, in the other dump it was already gone. The error code was the same: 14001, 0x36b1 in hex. Some details of the dumps:

FAILURE_BUCKET_ID: X64_0xF4_36b1_IMAGE_wininit.exe

0: kd> knL
 # Child-SP RetAddr Call Site
00 fffff880`03581b08 fffff800`01e5fab2 nt!KeBugCheckEx
01 fffff880`03581b10 fffff800`01e0aabb nt!PspCatchCriticalBreak+0x92
02 fffff880`03581b50 fffff800`01d89f04 nt! ?? ::NNGAKEGL::`string'+0x17486
03 fffff880`03581ba0 fffff800`01ad0e53 nt!NtTerminateProcess+0xf4
04 fffff880`03581c20 00000000`778d157a nt!KiSystemServiceCopyEnd+0x13
05 00000000`0020f7a8 00000000`00000000 0x778d157a

0: kd> !error 36b1
Error code: (Win32) 0x36b1 (14001) - The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.

The issue in this case occurs because WinPE tries to compact the offline registry and fails to commit the registry hives back to disk. This problem only happen when you deploy windows 7 and use WinPE 5.0 or 5.1, 32 bit, to deploy the image. Resolution is to set this registry value in the boot.wim using DISM:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Configuration Manager

Name: RegistryReorganizationLimitDays
Datatype: DWORD
Value: 365

Setting this value has the effect that the registry hives are not compacted as long as the modified date of the hives is not older than a year. When you intend to use the deployment longer than a year,  a higher value must be chosen.

If you have run into this issue and you managed to resolve it using this information, please let me know! Do note that not all STOP 0xF4 errors are the same! If you have another 0xF4 and need help, reach out!

Additional

We have had several customer run into this issue and found that just setting that registry key does not always correct the problem.

Basically Windows 8.1 adds a feature when mounting the registry that compacts it (gets rid of unused space) when loading it for the first time. During the Apply Operating System and Driver tasks, the Task Sequence mounts the registry. Since the Task Sequence is in WinPE 5 (which is based off of Windows 8.1) during these mounts, the Windows 7 registry, which has never been compacted before, gets compacted. When the registry is loaded to do the compacting, it needs a piece of contiguous memory that is as big as the registry to do the compacting. The size of the registry can vary a lot depending on the image, but you can say that it will usually be around 60MB in size.

This process is sometimes a problem when using an x86 boot image. Basically because WinPE  has no page file, WinPE relies completely on what physical memory is available. The end result is that memory in WinPE gets very fragmented. With x64 WinPE this is not a big deal on most modern PCs because there is a lot of memory available. However with x86 boot images, there is a limit of how much memory can be addressed (around 3GB) regardless of how much memory the PC actually has. The end result is that when the memory gets very fragmented with an x86 boot image, there is less of a chance of there being a piece of contiguous memory that large enough to hold the registry that needs to be mounted. In these situations, when the registry gets mounted, it gets corrupted and part of the registry basically gets chopped off.

The result of the registry corruption can be seen as blue screens (like the customer is seeing) or other symptoms like Windows Update not working. It really depends on what got chopped out of the registry.

There are several solutions to the problem:

1) Use an x64 boot image. Because x64 boot images can address a much larger amount of memory, the chance of there not being a piece of contiguous memory that is large enough to hold the registry is extremely low. This will eliminate the problem from every happening

2) Prevent the registry from compacting in WinPE 5:
a. On an x64 Windows 8.1 PC, mount the Windows 7 OS WIM and then load the SYSTEM and SOFTWARE registry hives from the image. This will initiate the initial compact of the registry. Unload the registry and it will save compacted. Unmount and commit the changes to the Windows 7 OS WIM file. This will prevent the initial compacting of the registry in WinPE 5.

b. Set the following registry keys on the WinPE 5 boot image. This needs to be done to the offline WinPE 5 boot image and cannot be done “real time” during the ConfigMgr Task Sequence:

HKLM\CurrentControlSet\Control\Session Manager\Configuration Manager
DWord : RegistryReorganizationLimitDays = 3650
DWord : RegistryReorganizationLimit = 536870912

This will cause subsequent registry loads not to compact for 10 years after the last compact. Additionally it will not compact the registry when the registry is under 500MB. Values are purposely set high to prevent registry compact from ever occurring.

Please note that “Configuration Manager” here does not refer to System Center Configuration Manager. It is a Windows key that exists whether or not System Center Configuration Manager is being used and is not related to System Center Configuration Manager

3) Use a WinPE 3.1 boot image instead. WinPE 3.1, which is based on Windows 7, does not attempt to compact the registry. Instructions for creating a proper WinPE 3.1 boot image for use in ConfigMgr 2012 R2 are at https://technet.microsoft.com/en-us/library/dn387582.aspx.

4) It appears that the bug that is causing this issue has been fixed in Windows 10. However it was just fixed recently (May) so we are unsure if it is going to make it in the final release of Windows 10. If it does make it in the final release of Windows 10, then an additional option is that when the final version of the Windows 10 ADK is released, upgrade to SP1 for ConfigMgr 2012 R2 and then use a WinPE 10 boot image.