This is the last post I will be writing on this blog. On 28th July 2015 I will leave Microsoft.

Thank you, all who have read my posts, contacted me, and who worked with me on resolving some of the issue seen on this blog.

All the best,
\Rob

150516: Finally: https://support.microsoft.com/en-us/kb/3055343 has the updated version. Please install at earliest convenience!

150512: to all affected, please bear with us a few more days, until the new release is out. This should be later this week, or early next week. As soon as the hotfix is available, I will update this post.

150501: Thanks to all of you who sent me additional dumps for this. There have been additional releases of netcfgx.dll, for example in 3021052, and as part of the March Rollup. This explains why some of you started seeing this issue after March/April updates. Checking internally shows that the various issues seen in the dumps provided should be resolved with the May (REL15-05) release of netcfgx.dll. The root cause of the problem allows for this issue to exhibit itself in many different froms, which explains the various dumps and workarounds found by some of you. Stay tuned for the mid-May release! I will update here once it's available.

150429: one of my readers sent me that they had 0x139s after the April updates, like some of you are seem to be having. They appear to have resolved it by recreating their virtual switch. I suggest that if applicable you also try this step and share results.

150417: I only mentioned the workaround (uninstall KB2966087) below in the comments. Adding it here now. Note that this fix was encompassed in KB2975719, so if you do not have the first one installed check for the latter, which is the August Rollup.

150415: one of the blog readers sent me a minidump of a STOP 0x139 on Windows Server 2012 that might be confused with this issue. Do note that the issue below is specific to 2012R2. The 2012 issue was:

2883658 "0x00000139" Stop error on a Windows Server 2012-based computer that has many TCP connections
https://support.microsoft.com/kb/2883658/EN-US

Recently we're seeing quite a few STOP 0x139/0xD1 errors, where the 0x139 stack is below:

00 ffffd000`748c5fc8 fffff803`7b3d14e9 nt!KeBugCheckEx
01 ffffd000`748c5fd0 fffff803`7b3d1810 nt!KiBugCheckDispatch+0x69
02 ffffd000`748c6110 fffff803`7b3d0a34 nt!KiFastFailDispatch+0xd0
03 ffffd000`748c62f0 fffff800`6a2ee3b1 nt!KiRaiseSecurityCheckFailure+0xf4
04 ffffd000`748c6480 fffff800`6a209308 NDIS!ndisNsiEnumerateAllInterfaceInformation+0x24c51
05 ffffd000`748c65a0 fffff800`6b264fc1 NETIO!NsiEnumerateObjectsAllParametersEx+0x20d
06 ffffd000`748c6790 fffff800`6b264bea nsiproxy!NsippEnumerateObjectsAllParameters+0x201
07 ffffd000`748c6980 fffff803`7b6223ef nsiproxy!NsippDispatch+0x5a
08 ffffd000`748c69c0 fffff803`7b62198e nt!IopXxxControlFile+0xa4f
09 ffffd000`748c6b60 fffff803`7b3d11b3 nt!NtDeviceIoControlFile+0x56
0a ffffd000`748c6bd0 00007ff9`3d890dfa nt!KiSystemServiceCopyEnd+0x13

The issue is caused by a problem in netcfgx.dll. We currently have an internal repro, and we expect to resolve this in REL15-05 timeframe.

Note that this issue could also manifest itself as a STOP 0x3B.

As always, reach out when you have this issue!

150312: Apologies for not updating this... the public release of this has slipped. The KB including this fix is https://support2.microsoft.com/kb/3031598.

150115: The hotfix is scheduled for release in the February 2015 cycle.

Hi All, back again after having been away for a long time... This time with a networking-related bugcheck, with the stack below. This issue is caused by a problem in MsLbfoProvider, but it is only triggered when there are underlying problems with the miniport (NIC). Specifically, there must have been a condition that triggered NDIS to automatically reset the miniport. After that, LBFO loses track of the VMQ queues assigned to the miniport, eventually causing the STOP 0xD1. I have filed a hotfix request for this issue today. Once I get updated information on if/when this is fixed, I'll update this post. If you think you are hit by this issue, open a case with support and let me know!

AV_MsLbfoProvider!vmqcGetFirstMappedMNic

 # Child-SP          RetAddr           Call Site
00 ffffd001`48cab918 fffff801`b17e47e9 nt!KeBugCheckEx
01 ffffd001`48cab920 fffff801`b17e303a nt!KiBugCheckDispatch+0x69
02 ffffd001`48caba60 fffff801`3e16f74f nt!KiPageFault+0x23a
03 ffffd001`48cabbf8 fffff801`3e16ee1e MsLbfoProvider!vmqcGetFirstMappedMNic+0xf
04 ffffd001`48cabc00 fffff801`3c621a13 MsLbfoProvider!LbfoReceiveNetBufferListsComplete+0x92
05 ffffd001`48cabc50 fffff801`3c611c2d NDIS! ?? ::FNODOBFM::`string'+0x99f5
06 ffffd001`48cabce0 fffff801`3c611cf2 NDIS!ndisReturnNetBufferListsInternal+0x12d
07 ffffd001`48cabd40 fffff801`3daefa01 NDIS!NdisReturnNetBufferLists+0x72
08 ffffd001`48cabda0 fffff801`3daef36b vmswitch!VmsPtNicPvtPacketRouted+0x1b1
09 ffffd001`48cabe30 fffff801`3daef03e vmswitch!VmsExtIoPacketRouted+0x28b
0a ffffd001`48cabee0 fffff801`3c618245 vmswitch!VmsExtPtSendNetBufferListsComplete+0x9e
0b ffffd001`48cabf30 fffff801`3c618a68 NDIS!ndisMSendCompleteNetBufferListsInternal+0x135
0c ffffd001`48cac000 fffff801`3daef3a5 NDIS!NdisMSendNetBufferListsComplete+0x2b8
0d ffffd001`48cac170 fffff801`3daeef62 vmswitch!VmsExtIoPacketRouted+0x2c5
0e ffffd001`48cac220 fffff801`3c621a8e vmswitch!VmsExtMpReturnNetBufferLists+0x42
0f ffffd001`48cac250 fffff801`3c611c2d NDIS! ?? ::FNODOBFM::`string'+0x9a70
10 ffffd001`48cac2e0 fffff801`3c611cf2 NDIS!ndisReturnNetBufferListsInternal+0x12d
11 ffffd001`48cac340 fffff801`3daef3ed NDIS!NdisReturnNetBufferLists+0x72
12 ffffd001`48cac3a0 fffff801`3daeeddc vmswitch!VmsExtIoPacketRouted+0x30d
13 ffffd001`48cac450 fffff801`3db03fad vmswitch!VmsNblHelperRefCountDecrementMany+0x7c
14 ffffd001`48cac490 fffff801`3db00d8a vmswitch!VmsRouterDeliverNetBufferLists+0x62d
15 ffffd001`48cac570 fffff801`3c611903 vmswitch!VmsExtPtReceiveNetBufferLists+0x13a
16 ffffd001`48cac5d0 fffff801`3c62129b NDIS!ndisMIndicateNetBufferListsToOpen+0x123
17 ffffd001`48cac690 fffff801`3c6128f6 NDIS!ndisMTopReceiveNetBufferLists+0x2db
18 ffffd001`48cac720 fffff801`3daed9c9 NDIS!NdisMIndicateReceiveNetBufferLists+0xb96
19 ffffd001`48cac900 fffff801`3c617a21 vmswitch!VmsExtMpSendNetBufferLists+0x359
1a ffffd001`48caca90 fffff801`3daf3347 NDIS!NdisSendNetBufferLists+0x551
1b ffffd001`48cacc80 fffff801`3daf2e14 vmswitch!VmsExtPtRouteNetBufferLists+0x377
1c ffffd001`48cacd50 fffff801`3c611903 vmswitch!VmsPtNicReceiveNetBufferLists+0x3c4
1d ffffd001`48caceb0 fffff801`3c62129b NDIS!ndisMIndicateNetBufferListsToOpen+0x123
1e ffffd001`48cacf70 fffff801`3c6128f6 NDIS!ndisMTopReceiveNetBufferLists+0x2db
1f ffffd001`48cad000 fffff801`3da687f0 NDIS!NdisMIndicateReceiveNetBufferLists+0xb96
20 ffffd001`48cad1e0 fffff801`3da6821a NdisImPlatform!implatTryToIndicateReceiveNBLs+0x1e8
21 ffffd001`48cad250 fffff801`3c611903 NdisImPlatform!implatReceiveNetBufferLists+0x1a2
22 ffffd001`48cad2d0 fffff801`3c6126dd NDIS!ndisMIndicateNetBufferListsToOpen+0x123
23 ffffd001`48cad390 fffff801`3dbc15fa NDIS!NdisMIndicateReceiveNetBufferLists+0x97d
24 ffffd001`48cad570 fffff801`3dbc2088 ixn64x64+0x295fa
25 ffffd001`48cad5c0 fffff801`3dba4607 ixn64x64+0x2a088
26 ffffd001`48cad640 fffff801`3dba4a01 ixn64x64+0xc607
27 ffffd001`48cad6b0 fffff801`3dba3ec8 ixn64x64+0xca01
28 ffffd001`48cad720 fffff801`3c613005 ixn64x64+0xbec8
29 ffffd001`48cad760 fffff801`b16c8130 NDIS!ndisInterruptDpc+0x1b5
2a ffffd001`48cad860 fffff801`b16c734b nt!KiExecuteAllDpcs+0x1b0
2b ffffd001`48cad9b0 fffff801`b17dc7ea nt!KiRetireDpcList+0xdb
2c ffffd001`48cadc60 00000000`00000000 nt!KiIdleLoop+0x5a

Update 150630: one of my colleagues mailed me additional info, see below under Additional.

Recently I came across two very similar STOP 0xF4 cases, sent to me by ConfigMgr colleagues. When examining the dump, it became clear there was a problem with services.exe. In one dump it was still present while exiting, in the other dump it was already gone. The error code was the same: 14001, 0x36b1 in hex. Some details of the dumps:

FAILURE_BUCKET_ID:  X64_0xF4_36b1_IMAGE_wininit.exe

0: kd> knL
 # Child-SP          RetAddr           Call Site
00 fffff880`03581b08 fffff800`01e5fab2 nt!KeBugCheckEx
01 fffff880`03581b10 fffff800`01e0aabb nt!PspCatchCriticalBreak+0x92
02 fffff880`03581b50 fffff800`01d89f04 nt! ?? ::NNGAKEGL::`string'+0x17486
03 fffff880`03581ba0 fffff800`01ad0e53 nt!NtTerminateProcess+0xf4
04 fffff880`03581c20 00000000`778d157a nt!KiSystemServiceCopyEnd+0x13
05 00000000`0020f7a8 00000000`00000000 0x778d157a

0: kd> !error 36b1
Error code: (Win32) 0x36b1 (14001) - The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.

The issue in this case occurs because WinPE tries to compact the offline registry and fails to commit the registry hives back to disk. This problem only happen when you deploy windows 7 and use WinPE 5.0 or 5.1, 32 bit, to deploy the image. Resolution is to set this registry value in the boot.wim using DISM:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Configuration Manager

Name:  RegistryReorganizationLimitDays
Datatype: DWORD
Value:  365

Setting this value has the effect that the registry hives are not compacted as long as the modified date of the hives is not older than a year. When you intend to use the deployment longer than a year,  a higher value must be chosen.

If you have run into this issue and you managed to resolve it using this information, please let me know! Do note that not all STOP 0xF4 errors are the same! If you have another 0xF4 and need help, reach out!

Additional

We have had several customer run into this issue and found that just setting that registry key does not always correct the problem.

Basically Windows 8.1 adds a feature when mounting the registry that compacts it (gets rid of unused space) when loading it for the first time. During the Apply Operating System and Driver tasks, the Task Sequence mounts the registry. Since the Task Sequence is in WinPE 5 (which is based off of Windows 8.1) during these mounts, the Windows 7 registry, which has never been compacted before, gets compacted. When the registry is loaded to do the compacting, it needs a piece of contiguous memory that is as big as the registry to do the compacting. The size of the registry can vary a lot depending on the image, but you can say that it will usually be around 60MB in size.

This process is sometimes a problem when using an x86 boot image. Basically because WinPE  has no page file, WinPE relies completely on what physical memory is available. The end result is that memory in WinPE gets very fragmented. With x64 WinPE this is not a big deal on most modern PCs because there is a lot of memory available. However with x86 boot images, there is a limit of how much memory can be addressed (around 3GB) regardless of how much memory the PC actually has. The end result is that when the memory gets very fragmented with an x86 boot image, there is less of a chance of there being a piece of contiguous memory that large enough to hold the registry that needs to be mounted. In these situations, when the registry gets mounted, it gets corrupted and part of the registry basically gets chopped off.

The result of the registry corruption can be seen as blue screens (like the customer is seeing) or other symptoms like Windows Update not working. It really depends on what got chopped out of the registry.

There are several solutions to the problem:

1) Use an x64 boot image. Because x64 boot images can address a much larger amount of memory, the chance of there not being a piece of contiguous memory that is large enough to hold the registry is extremely low. This will eliminate the problem from every happening

2) Prevent the registry from compacting in WinPE 5:
a. On an x64 Windows 8.1 PC, mount the Windows 7 OS WIM and then load the SYSTEM and SOFTWARE registry hives from the image. This will initiate the initial compact of the registry. Unload the registry and it will save compacted. Unmount and commit the changes to the Windows 7 OS WIM file. This will prevent the initial compacting of the registry in WinPE 5.

b. Set the following registry keys on the WinPE 5 boot image. This needs to be done to the offline WinPE 5 boot image and cannot be done “real time” during the ConfigMgr Task Sequence:

HKLM\CurrentControlSet\Control\Session Manager\Configuration Manager
DWord : RegistryReorganizationLimitDays = 3650
DWord : RegistryReorganizationLimit = 536870912

This will cause subsequent registry loads not to compact for 10 years after the last compact. Additionally it will not compact the registry when the registry is under 500MB. Values are purposely set high to prevent registry compact from ever occurring.

Please note that “Configuration Manager” here does not refer to System Center Configuration Manager. It is a Windows key that exists whether or not System Center Configuration Manager is being used and is not related to System Center Configuration Manager

3) Use a WinPE 3.1 boot image instead. WinPE 3.1, which is based on Windows 7, does not attempt to compact the registry. Instructions for creating a proper WinPE 3.1 boot image for use in ConfigMgr 2012 R2 are at https://technet.microsoft.com/en-us/library/dn387582.aspx.

4) It appears that the bug that is causing this issue has been fixed in Windows 10. However it was just fixed recently (May) so we are unsure if it is going to make it in the final release of Windows 10. If it does make it in the final release of Windows 10, then an additional option is that when the final version of the Windows 10 ADK is released, upgrade to SP1 for ConfigMgr 2012 R2 and then use a WinPE 10 boot image.

///

Hi All,

In Support we frequently use memory dumps as a tool to troubleshoot problems, such as a hung or frozen machine. Sometimes however, capturing a dump proves anything but trivial, and it may take extended amounts of time and effort to finally get the data we want. Now, when working with virtual machines, I still see engineers struggle with capturing a memory dump of a hung vm. However, both Hyper-V and VMware provide functionality to capture the machine state and convert the state file(s) - vsv/bin on Hyper-V, vmss on VMware -, to a dump file that can be read into WinDbg. Converting is simple: on Hyper-V use vm2dmp, on VMware use vmss2core (thanks to Paudie from VMware for sharing this link with me).

Let me know if this bit of info helped you!

Note: should you want to go the "old" way of getting dumps on vms, then you'll surely appreciate the virtual NMI functionality in Hyper-V 2012 R2. ;)

\Rob

Hi All, recently we have been seeing an increase in STOP 0x133 (on non-clustered) and 0x9E (on clustered) cases, where the root cause is that there are too many NBLs indicated in a DPC. This causes the DPC watchdog timer to expire, resulting in mentioned STOP errors. Part of an example stack is

1f ffffd001`1d4f2800 fffff800`d9cfa347 NDIS!NdisSendNetBufferLists+0x551
20 ffffd001`1d4f29f0 fffff800`d9cf9e14 vmswitch!VmsExtPtRouteNetBufferLists+0x377
21 ffffd001`1d4f2ac0 fffff800`d845b903 vmswitch!VmsPtNicReceiveNetBufferLists+0x3c4
22 ffffd001`1d4f2c20 fffff800`d846b29b NDIS!ndisMIndicateNetBufferListsToOpen+0x123
23 ffffd001`1d4f2ce0 fffff800`d845c8f6 NDIS!ndisMTopReceiveNetBufferLists+0x2db
24 ffffd001`1d4f2d70 fffff800`d81df7f0 NDIS!NdisMIndicateReceiveNetBufferLists+0xb96
25 ffffd001`1d4f2f50 fffff800`d81df21a NdisImPlatform!implatTryToIndicateReceiveNBLs+0x1e8
26 ffffd001`1d4f2fc0 fffff800`d845b903 NdisImPlatform!implatReceiveNetBufferLists+0x1a2
27 ffffd001`1d4f3040 fffff800`d845c6dd NDIS!ndisMIndicateNetBufferListsToOpen+0x123
28 ffffd001`1d4f3100 fffff800`d89e9eb7 NDIS!NdisMIndicateReceiveNetBufferLists+0x97d

At present, there is no definitive solution available, but there are some mitigating actions available:

1. Optimize RSS/VMQ configuration. A NIC miniport indicating a >256 NBL chain per CPU indicates RSS/VMQ is not correctly used. See https://blogs.technet.com/b/networking/archive/2013/09/10/vmq-deep-dive-1-of-3.aspx.

2. Use a Receive Side Throttling (RST) capable NIC.

3 Reduce the number of Receive Buffers for the miniport indicating the high amount of NBLs. Note that this setting is not available for Hyper-V vNICs. Also, by setting this a performance limitation might be introduced. Monitor your network performance after changing this value.

Also ensure that NIC driver/firmware are up-to-date. Additional information: https://msdn.microsoft.com/en-us/library/windows/hardware/ff567241(v=vs.85).aspx.

When you think you are hitting this scenario, please reach out to us.

X64_0x3B_nt!ExDeferredFreePool+249

Other common pool-related STOP errors include 0x1E, 0x50, 0xA, and 0x8E.

Quite often, the stack in such cases reveals an offending (third party) driver. Also often however, there is no clear indication of what might have happened. Take note also, that simply having a driver on the stack does not make it the culprit of the crash: it could have simply tripped over a corruption that was caused by another driver at an earlier point in time. In most cases, one memory dump is not enough, and we need additional information in subsequent dumps to find the root cause of the problem. We can obtain such info by enabling Driver Verifier. The options to specify related to Pool issues are Special Pool and Pool Tracking, under the Standard Flags:

STANDARD FLAGS:
These flags are considered standard options for Driver Verifier and can be
set using '/standard' or by the combination of the options: '/flags 0x209BB'
  bit  0 (0x00000001) - Special pool
  bit  1 (0x00000002) - Force IRQL checking
  bit  3 (0x00000008) - Pool tracking
  bit  4 (0x00000010) - I/O verification
  bit  5 (0x00000020) - Deadlock detection
  bit  7 (0x00000080) - DMA checking
  bit  8 (0x00000100) - Security checks
  bit 11 (0x00000800) - Miscellaneous checks
  bit 17 (0x00020000) - DDI compliance checking

The syntax to enable this: "verifier /flags 0x9 /all". This needs a reboot to be effective. Should you want to enable this run-time: "verifier /volatile /flags 0x9 /adddriver *". No reboot needed, but verification will also not be persistent across reboots. Please refer to the Driver Verifer link above for further information.

As a bonus, when enabling Pool Tracking, we also enable TrackNblOwner, which comes in handy when debugging network problems, as my friend and colleague Karim explains here.

Pool Tracking creates an in-memory database to store thread stacks related to pool allocations. By default, this is 65536 (0x10000) entries, but in cases where this is not enough, we can allocate up to 1048576 entries (0x100000), through the VerifierPoolTracesLength registry value under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

Note: only Windows 8 and higher. Each database entry is 128 bytes, so allocating one million (1048576) of these, only costs us 128MB of physical RAM, as the stack trace database is in Nonpaged Pool.

Thoughts? Comments? Let me know!

315263 How to read the small memory dump file that is created by Windows if a crash occurs
https://support.microsoft.com/kb/315263/EN-US

\Rob