Critical_Structure_Corruption Windows Server 2012 R2 ntoskrnl.exe/ntkrnlmp.exe

Question

With regard to this question:

I'm experiencing this same issue on Windows Server 2012 R2.

All dmp files zipped and uploaded. You can download them from my SkyDrive**here.**

Please note that the 2012 R2 server is a VMware virtual machine that is experiencing this error.  The error began after installing VMware Tools on a fresh installation.

I may have resolved this already and determined what the culprit was but would like assistance in analyzing the dmp files I have provided for more clarity.

Answer 1

So running the 5.5.0a version stops the crashes?

The software I used was WinDbg. Analyzing .DMP files is a science really, and something that requires much reading and practice over time. Looking at the 'probably caused by' to find a culprit is the absolute beginning. You eventually learn to look and read through call stacks, memory, loaded module lists, what codes mean, etc. You'll also be able to look at a dump, its bug check, and even if there is no clear indication of what the issue is anywhere in the dump itself, you can from there form educated guesses based off certain 3rd party drivers you suspect causing conflicts, etc.

For your dumps, I took a look and saw that there was nothing really that implied hardware errors yet, but also the modules list noted the vmxnet3n61x64.sys driver being dated pre-2011 if I remember correctly, which said to me that it is possibly not the latest version depending. It was one of the only 3rd party drivers on the system given it's a server especially, so this raised more than a few flags to me.

Regards,

Patrick

Answer 2

Hi,

All of the attached DMP files are of the CRITICAL_STRUCTURE_CORRUPTION (109) bug check.

This indicates that the kernel has detected critical kernel code or data corruption.

There are generally two causes for this bug check:

1. A driver has inadvertently, or deliberately, modified critical kernel code or data. Microsoft Windows Server 2003 with Service Pack 1 (SP1) and later versions of Windows for x64-based computers do not allow the kernel to be patched except through authorized Microsoft-originated hot patches. For more information, see Patching Policy for x64-based Systems.
2. A hardware corruption occurred. For example, the kernel code or data could have been stored in memory that failed.

Looking at the dump initially, I can see no real clues as to what may be causing it. Most if not all *109's start like this. I took a look at the loaded modules list and found a driver that MAY be possibly causing this issues if device driver related - vmxnet3n61x64.sys (This files most often belongs to product VMware PCIe Ethernet Adapter NDIS 6.1 (64-bit). and were most often developed by company **VMware, Inc.)**If not, let's go ahead and run a Memtest for NO LESS than ~8 passes (several hours):

Memtest86+:

Download Memtest86+ here:

http://www.memtest.org/

Which should I download?

You can either download the pre-compiled ISO that you would burn to a CD and then boot from the CD, or you can download the auto-installer for the USB key. What this will do is format your USB drive, make it a bootable device, and then install the necessary files. Both do the same job, it's just up to you which you choose, or which you have available (whether it's CD or USB).

How Memtest works:

Memtest86 writes a series of test patterns to most memory addresses, reads back the data written, and compares it for errors.

The default pass does 9 different tests, varying in access patterns and test data. A tenth test, bit fade, is selectable from the menu. It writes all memory with zeroes, then sleeps for 90 minutes before checking to see if bits have changed (perhaps because of refresh problems). This is repeated with all ones for a total time of 3 hours per pass.

Many chipsets can report RAM speeds and timings via SPD (Serial Presence Detect) or EPP (Enhanced Performance Profiles), and some even support changing the expected memory speed. If the expected memory speed is overclocked, Memtest86 can test that memory performance is error-free with these faster settings.

Some hardware is able to report the "PAT status" (PAT: enabled or PAT: disabled). This is a reference to Intel Performance acceleration technology; there may be BIOS settings which affect this aspect of memory timing.

This information, if available to the program, can be displayed via a menu option.

Any other questions, they can most likely be answered by reading this great guide here:

http://forum.canardpc.com/threads/28864-FAQ-please-read-before-posting

If Memtest goes ~8 passes without error, we could enable Driver Verifier to look for further possible device driver conflicts and or corruption:

Driver Verifier:

What is Driver Verifier?

Driver Verifier is included in Windows 8, 7, Windows Server 2008 R2, Windows Vista, Windows Server 2008, Windows 2000, Windows XP, and Windows Server 2003 to promote stability and reliability; you can use this tool to troubleshoot driver issues. Windows kernel-mode components can cause system corruption or system failures as a result of an improperly written driver, such as an earlier version of a Windows Driver Model (WDM) driver.

Essentially, if there's a 3rd party driver believed to be at issue, enabling Driver Verifier will help flush out the rogue driver if it detects a violation.

Before enabling Driver Verifier, it is recommended to create a System Restore Point:

Vista - START | type rstrui - create a restore point

Windows 7 - START | type create | select "Create a Restore Point"

Windows 8 - http://www.eightforums.com/tutorials/4690-restore-point-create-windows-8-a.html

How to enable Driver Verifier:

Start > type "verifier" without the quotes > Select the following options -

1. Select - "Create custom settings (for code developers)"
2. Select - "Select individual settings from a full list"
3. Check the following boxes -

• Special Pool
• Pool Tracking
• Force IRQL Checking
• Deadlock Detection
• Security Checks (Windows 7 & 8)
• DDI compliance checking (Windows 8)
• Miscellaneous Checks

4. Select  - "Select driver names from a list"
5. Click on the "Provider" tab. This will sort all of the drivers by the provider.
6. Check EVERY box that is [B]NOT[/B] provided by Microsoft / Microsoft Corporation.
7. Click on Finish.
8. Restart.

Important information regarding Driver Verifier:

• If Driver Verifier finds a violation, the system will BSOD.
• After enabling Driver Verifier and restarting the system, depending on the culprit, if for example the driver is on start-up, you may not be able to get back into normal Windows because Driver Verifier will flag it, and as stated above, that will cause / force a BSOD.

If this happens, do not panic, do the following:

• Boot into Safe Mode by repeatedly tapping the F8 key during boot-up.
• Once in Safe Mode - Start > type "system restore" without the quotes.
• Choose the restore point you created earlier.

If you did not set up a restore point, do not worry, you can still disable Driver Verifier to get back into normal Windows:

• Start > Search > type "cmd" without the quotes.
• To turn off Driver Verifier, type in cmd "verifier /reset" without the quotes.

・    Restart and boot into normal Windows.

How long should I keep Driver Verifier enabled for?

It varies, many experts and analysts have different recommendations. Personally, I recommend keeping it enabled for at least 24 hours. If you don't BSOD by then, disable Driver Verifier.

My system BSOD'd, where can I find the crash dumps?

They will be located in %systemroot%\Minidump

Any other questions can most likely be answered by this article:

http://support.microsoft.com/kb/244617

Regards,

Patrick

Answer 3

My pleasure, thanks very much for the information. It's appreciated as always just in case I run into a similar case.

If and when you are comfortable and feel your issue has been solved, I'd recommend marking the post of mine that answered your question so this thread no longer shows up as requiring an answer.

Regards,

Patrick

Answer 4

OK, great to know.  I wasn't very far off then from being able to determine that stuff myself as well.  I am just not very familiar with the debugging tools yet and hope to familiarize myself more.

To answer your question, yes, running the OS on the newer 5.5 version did indeed resolve the crashing I was experiencing when running in the 5.0 version.  5.5 provides full support and compatibility for this operating system where as the 5.0 version doesn't.  Although it did allow me to install and run the OS in a VM on the older version, it proved to be unstable and would crash every 15 minutes or so.  Since running in the 5.5 environment, I haven't experienced a single crash on this system.

Thank you for taking the time to assist me and answer my questions

Answer 5

Patrick,

Thank your for all of the helpful and detailed information.  The vmxnet3 driver is indeed a VMware specific network driver but it is not causing the problem as the issue persisted even after removal/uninstallation of VMware Tools which then removes said driver.

I took further initiative to try to recreate the issue in a Dev/Lab environment where I am running the newest release of VMware vSphere 5.5 utilizing vCenter 5.5.0a as well and the issue no longer occurred.

The environment where I was experiencing this issue was in a VMware vSphere 5.0 environment utilizing vCenter 5.0U1 which I firmly believe does not fully support Windows Server 2012/R2.

At the time of it's release, it may have only supported the release preview of Windows Server 2012 as the actual OS had yet to be released. It lists the OS choice as "Windows Server 8".  vSphere/vCenter 5.5 has full support for Windows Server 2012/R2 and so far I have no longer had any further issues.

I did try to analyze the dmp's myself a bit with WinDbg but would often see that the "Analysis was Inconclusive".  I am not sure if I had all of the correct symbols installed or what not which then prompted me to start this thread.

Would you be able to tell me what you used and how you used it as well as what was done/used to determine the vmxnet3 adapter as a "possible" culprit?

Thanks in advance!