[RESOLVED] Win2008R2 SP1: STOP 0x19 in volsnap!VspFreeBitMap+3d

Status: Resolved.

Update 110606: We have determined that the STOP 0x19 has the same root cause as the STOP 0x7E described in KB2460912. This leaves us with the STOP 0x7E of Update 110527, for which I will create a new post.

Update 110527: Another customer sent me a dump of a Win2008R2 machine experiencing a STOP 0x7E (KB2460912 is installed), with the following stack:

0: kd> knL
 # Child-SP RetAddr Call Site
00 fffff880`021dc068 fffff800`016f26ab nt!memset+0x80
01 fffff880`021dc070 fffff880`01ac0c0b nt!RtlClearBits+0x8b
02 fffff880`021dc0a0 fffff880`01ac50b9 volsnap!VspQueryNtfsFreeSpaceBitmap+0x2fb
03 fffff880`021dc170 fffff880`01ac5ce6 volsnap!VspMarkFreeSpaceInBitmap+0x179
04 fffff880`021dc360 fffff880`01ac7e21 volsnap!VspOptimizeDiffAreaFileLocation+0x2a6
05 fffff880`021dc6e0 fffff880`01ad769d volsnap!VspOpenDiffAreaFile+0x481
06 fffff880`021dc860 fffff880`01ade5f7 volsnap!VspCreateInitialDiffAreaFile+0x1ed
07 fffff880`021dc8b0 fffff880`01adf5e6 volsnap!VspTryPrepareForSnapshot+0x737
08 fffff880`021dcb90 fffff880`01abd0fc volsnap!VspPrepareForSnapshot+0x116
09 fffff880`021dcc50 fffff800`019c80c3 volsnap!VspPostWorker+0x2c
0a fffff880`021dcc80 fffff800`016dc021 nt!IopProcessWorkItem+0x23
0b fffff880`021dccb0 fffff800`0196e32e nt!ExpWorkerThread+0x111
0c fffff880`021dcd40 fffff800`016c3666 nt!PspSystemThreadStartup+0x5a
0d fffff880`021dcd80 00000000`00000000 nt!KxStartSystemThread+0x16

The FailureBucket is X640x7E_volsnap!VspQueryNtfsFreeSpaceBitmap+2fb. I've asked for a kernel memory dump since I only got a minidump of this. When that comes in and it is indeed another issue, I'll create a separate post on it.

Update 110517-2: The STOP 0x50 is indeed resolved by 2460912. Reason that 2505454 did not fix this, is that customer was running the GDR version which does not include all hotfixes. Resolution for the STOP 0x50 was: uninstall 2505454, install 2460912, reinstall 2505454 (now the LDR version will be on the machine). The STOP 0x19 is still under investigation.

Update 110517-1: Yesterday late afternoon I received another dump of one of the customers, which actually showed a somewhat different stack:

4: kd> knL
 # Child-SP RetAddr Call Site
00 fffff880`023d1068 fffff800`02ef80bb nt!memset+0x9e
01 fffff880`023d1070 fffff880`01b3ccf2 nt!RtlClearBits+0x8b
02 fffff880`023d10a0 fffff880`01b40cf9 volsnap!VspQueryNtfsFreeSpaceBitmap+0x2a2
03 fffff880`023d1170 fffff880`01b41926 volsnap!VspMarkFreeSpaceInBitmap+0x179
04 fffff880`023d1360 fffff880`01b43fb1 volsnap!VspOptimizeDiffAreaFileLocation+0x2a6
05 fffff880`023d16e0 fffff880`01b5382d volsnap!VspOpenDiffAreaFile+0x481
06 fffff880`023d1860 fffff880`01b5a787 volsnap!VspCreateInitialDiffAreaFile+0x1ed
07 fffff880`023d18b0 fffff880`01b5b776 volsnap!VspTryPrepareForSnapshot+0x737
08 fffff880`023d1b90 fffff880`01b3922c volsnap!VspPrepareForSnapshot+0x116
09 fffff880`023d1c50 fffff800`031cdf33 volsnap!VspPostWorker+0x2c
0a fffff880`023d1c80 fffff800`02ee1a21 nt!IopProcessWorkItem+0x23
0b fffff880`023d1cb0 fffff800`03174cce nt!ExpWorkerThread+0x111
0c fffff880`023d1d40 fffff800`02ec8fe6 nt!PspSystemThreadStartup+0x5a
0d fffff880`023d1d80 00000000`00000000 nt!KxStartSystemThread+0x16

This closely resembles the problem described in:

2460912 "0x0000007E" Stop error when you create snapshots of a volume in Windows Server 2008 R2 or in Windows 7
https://support.microsoft.com/default.aspx?scid=kb;en-US;2460912

The only thing is, that the customer is running with volsnap.sys of this KB, which is newer:

2505454 The startup process is delayed on a computer that has a large hard disk installed and is running Windows 7 or Windows Server 2008 R2
https://support.microsoft.com/default.aspx?scid=kb;en-US;2505454

We are further investigating this. Updates will follow shortly.

Update 110514: We now have two customer cases of this. Further research on the dumps is scheduled for Monday.

Update 110512: another customer mailed me they are hitting this problem using the link below. A verifier-enabled dump did not yield additional information, but I am engaging internal resources to further follow up on this. Stay tuned!

My colleague Shasank submitted me a new dump, that looks to be caused by an issue in volsnap.sys. This is yet to be confirmed though, but I just wanted to put a heads-up here, in case anyone out there also has come across this... If so, then please send me a mail telling me you have this issue.

The bugcheck of this particular issue is:

BugCheck 19, {21, fffffa800de58000, 68d0, 0}

The crashing stack is:

4: kd> knL
 # Child-SP RetAddr Call Site
00 fffff880`023c3088 fffff800`026019b2 nt!KeBugCheckEx
01 fffff880`023c3090 fffff880`01a94dfd nt!ExFreePoolWithTag+0xffffffff`ffffec22
02 fffff880`023c3140 fffff880`01aad117 volsnap!VspFreeBitMap+0x3d
03 fffff880`023c3170 fffff880`01aadcd6 volsnap!VspMarkFreeSpaceInBitmap+0x1e7
04 fffff880`023c3360 fffff880`01aafe11 volsnap!VspOptimizeDiffAreaFileLocation+0x2a6
05 fffff880`023c36e0 fffff880`01abf68d volsnap!VspOpenDiffAreaFile+0x481
06 fffff880`023c3860 fffff880`01ac65e7 volsnap!VspCreateInitialDiffAreaFile+0x1ed
07 fffff880`023c38b0 fffff880`01ac75d6 volsnap!VspTryPrepareForSnapshot+0x737
08 fffff880`023c3b90 fffff880`01aa50fc volsnap!VspPrepareForSnapshot+0x116
09 fffff880`023c3c50 fffff800`027cdf33 volsnap!VspPostWorker+0x2c
0a fffff880`023c3c80 fffff800`024e1a21 nt!IopProcessWorkItem+0x23
0b fffff880`023c3cb0 fffff800`02774cce nt!ExpWorkerThread+0x111
0c fffff880`023c3d40 fffff800`024c8fe6 nt!PspSystemThreadStartup+0x5a
0d fffff880`023c3d80 00000000`00000000 nt!KxStartSystemThread+0x16

The problem here is that an array of bitmaps has been corrupted. In the dump I checked, the sixth bitmap is corrupted:

4: kd> dq 0xfffffa80`14787450
fffffa80`14787450 00000000`00080000 00000000`00000000
fffffa80`14787460 00000001`00080000 00000000`00000000
fffffa80`14787470 00000000`00080000 00000000`00000000
fffffa80`14787480 00000000`00080000 00000000`00000000
fffffa80`14787490 00000000`00080000 00000000`00000000
fffffa80`147874a0 47cbbbaa`00080000 00000000`00000000 // the corrupted bitmap is here
fffffa80`147874b0 00000000`00080000 00000000`00000000
fffffa80`147874c0 00470045`00080000 00000000`00000000

As said, please mail me when you have this issue too, so we can further investigate. In this particular case, we are going to enable "verifier /volatile /flags 0x9 /adddriver volsnap.sys" to gather more information on this.

Watch this space for further updates.