My customer scenario:
Customer creates a task sequence to deploy, patch and capture a custom wim for deployment which succeeds. They then deploy patched wim and find that KB2975719 is found to be applicable and is installed adding roughly 10 minutes to overall build process. My customer wanted to know why the patch was not included in the original capture as it should have been.

Root cause:
The patch has six child updates, five of which have applicability rules. The first five are satisfied on the first run, the sixth with child update ID {89F63D7F-A24F-42AB-B091-805F7D18136B}.204 has a requirement that Win32k.sys be greater than certain version to be installed. This is not satisfied on the first run so it’s not installed. On the second run Win32k.sys has the correct version and the patch is installed.  The ZTIWindowsUpdate.wsf script has a limitation where it will not attempt to install the same package\payload two times which is a requirement for this kind of patch. As a result, the patch is not installed as expected in image creation.  Deployment of the patch with WSUS\SCCM\Windows Update after the OS is deployed will succeed as expected because the prerequisites are there.

3 possible Workarounds to include the patch in reference image:

#1 Download a revised version of ZTIWindowsUpdate.wsf which allows the patch to be installed as desired. Simple download from the following blog posting. (This is preferred as this does not require the customer to make any changes to their task sequence.)
#2 Add a step into the task sequence which will reset the listing of Windows Update patches found. In this way, ZTIWindowsUpdate.wsf will detect KB2975719 and it will be installed. This is documented in the following blog. (This will require a change to task sequence)
#3 Use the LTISuspend.wsf feature to pause the deployment and manually install the patch, click the resume shortcut on the desktop to resume capture. This is documented in the following blog.

A bug has been filed with the MDT team for consideration
https://connect.microsoft.com/ConfigurationManagervnext/feedback/details/1009473

** Updated 1/8/2015 **

Microsoft has refreshed installation media for Windows 8.1 Update and 2012 R2 Update from MSDN and VLSC download locations to include November 2014 cumulative rollup.  Please use that media for deploying and capturing reference images. 

What is an example of a label name and what is it?  From a Windows internals perspective, the label name is stored per volume in a NTFS metafile called $Volume.  A file record within exists called $Volume_Name that contains the label name.  Example from UI where ‘Demo’ is the volume label assigned:

Based on the definition above, when accessing a share remotely, you do not have block level access, the data is being processed through a network file redirector on your behalf.  In this case, the Windows member server running the client application is merely the client receiving this information over the network. 

For me, the main questions were:

1. Does this work as expected taking 3rd party appliance out of the equation?

2. Is the Windows 2008 R2 server receiving the expected data but not rendering it in the UI or is the Windows 2008 R2 server not receiving expected data from 3rd party filer at all?

In my lab I quickly mocked up 3 scenarios as to compare some of the legacy operating systems to current operating system for any differences simply using Windows to Windows.  Can we reproduce the issue without 3rd party?

No.  When using Windows as the server sharing the volume, the volume label was always present on the client.  From the customer perspective, this was a Windows problem for sure.  How could it not be since the Windows 2003 server displayed the label name and Windows 2008 R2 did not?

My next step was to collect a network trace of the working and nonworking scenario with hopes this would provide a clue on root cause. I collected a ‘working’ trace using a new protocol analyzer called (MMA)Microsoft Message Analyzer which has replaced Netmon.

Alternatively you can collect a trace with Netsh and ETW provider in the event you can’t or don’t want to install MMA on the system you’re investigating.

ipconfig /flushdns
klist purge
netsh trace start capture = yes

<repro>
netsh trace stop

This will create a ETL file that you can consume with MMA.  You can also export the trace into a .cap or .pcap in case you want to use Netmon or WireShark.  Current versions of the OS like 2012 R2 allow remote captures with MMA but that’s for another blog  

Now that we have the trace to review, leverage a feature in MMA I really like is called protocol dashboard which shows a summary of all protocols in the trace.  Since we are mapping a drive we know this should be a SMB or Server Message Block conversation.  Our working baseline looks like this:

The primary protocol used for the working scenario was SMB.  You can access this feature “Protocol Dashboard” by right clicking on the trace and selecting “New viewer”…

We should document was version of SMB works and verify that label name is received.  Using a filter (SMB.Summary contains "Negotiate") I’m able to quickly see the following:

What does NTLM0.12 mean?  Fortunately reference material Definition of SMB version numbers defines it to mean SMBv1 for Windows XP\2003.  This must mean that “SMB” in the protocol summary with MMA is really SMBv1.  Since our working scenario is on a Windows 2003 server this makes sense.  Finally, where is the label name in the working trace?  Open the trace with MMA

*IPs are from private nonroutable network

* Due to real estate and formatting you may need to click on the picture to see image with best quality

Now we need to add a filter to locate the Label name.  I’ve found that by typing in SMB then DOT pops the intellisense feature.. SMB.V will pop a list of methods to choose from, one of which is SMB.VolumeLabel.  Inserting that into filter, click apply makes the trace much easier to review. 

Let’s click on message 539 and see the request\response.

Expand the response message until you see the label name.  Demo is the string Windows received from the target hosting the share.  Now we have the signature of what to look for.  If we look at the broken trace and see the VolumeLabel populated then this is a Windows bug.  If we don’t receive the value then it’s a bug with the 3rd party appliance.  One additional tip, if you right click on the VolumeLabel item a fly out window pops up, select “Add as Analysis Grid Column” which will add a column which makes it easier to see.

Let’s open the broken trace and see what’s different including the new column…

No messages are shown.  Why?  Let’s take a look at the protocol dashboard for the broken trace to understand why.  Working is on the left and broken on the right.  This will lead to the first break in the case.  The working trace is using SMBv1 and broken is SMBv2 which also lines up with the Operating System.  Windows 2003 SMBv1 and 2008 R2 SMBv2.

Let’s confirm by comparing the dialect negotiation as we did earlier.  Confirmed SMB2, earlier it was NTLM0.12.

Let’s change our filter from SMB.VolumeLabel to SMB2.VolumeLabel.  This is the customer trace so I’ve remove everything except the string we are interested in. 

Our answer is in the absence of data.  The fact is the 3rd party appliance is not providing the VolumeLabel to Windows and therefore we are not displaying it.  Our options…

1. Fix the application to remove dependency on label name

2. File (DCR) Design Change Request with 3rd party appliance to include label name over SMB2 just like they did in SMBv1

3. (Not recommended) Follow the following KB and force SMBv1 on Windows 2008 R2

How to enable and disable SMBv1, SMBv2, and SMBv3 in Windows Vista, Windows Server 2008, Windows 7, Windows Server 2008 R2, Windows 8, and Windows Server 2012
https://support.microsoft.com/kb/2696547

I hope you found the blog interesting and hope you give Message Analyzer a try for future protocol analysis.

What changed? Driver related? Not enough memory? Bad DNS?

The customer did mention their goal was to move to a new environment where DHCP\PXE would not be available and at least that was one known variable.  The next step is to carefully consider the boot process and where logs which would be required to move forward would be located.  This is a well-known article which talks about where setup logs may reside based on the phase of the deployment… but in this case the investigation is before Windows setup even starts.  So where are the logs in Windows Preinstallation Environment (Windows PE) phase?

The X: drive represents the RAMDisk environment for PE.  Typically, we need the content from the minint directory but for this scenario we’re going to focus on the Windows PE initialization log and unattend.xml.

X:\windows\system32\wpeinit.log
X:\unattend.xml

The wpeinit.log should shed some light on the earliest phase of the boot process where the delay has been reported.  Why do we even need an unattend for PE?  A mechanism is required to bootstrap MDT from within Windows PE and the unattend provides that capability by calling wscript.exe X:\Deploy\Scripts\LiteTouch.wsf. 

In the first scenario, we collected logs from a legacy environment where performance meets customer expectations.

[Scenario #1 wpeinit.log DHCP Enabled on subnet – Default configuration]

Total 1 second to install network driver and verify via WMI that address is assigned.  Very fast.
2014-08-13 18:15:55.935, Info      Spent 1187ms installing network components
2014-08-13 18:15:56.420, Info      Installing device vmbus\{f8615163-df3e-46c5-913f-f2d2f965ed0e} X:\windows\INF\wnetvsc.inf succeeded
2014-08-13 18:15:56.576, Info      Installing device root\kdnic X:\windows\INF\kdnic.inf succeeded
2014-08-13 18:15:56.685, Info      Spent 750ms installing network drivers
2014-08-13 18:15:56.795, Info      QueryAdapterStatus: found operational adapter with DHCP address assigned.
2014-08-13 18:15:56.795, Info      Spent 0ms confirming network initialization; status 0x00000000

[Scenario #2 wpeinit.log DHCP Disabled on subnet – trying to improve as you can see delay below when timeout is reached]

Total 1.5 minutes
2014-08-13 18:18:54.685, Info      Spent 1078ms installing network components
2014-08-13 18:18:55.185, Info      Installing device vmbus\{f8615163-df3e-46c5-913f-f2d2f965ed0e} X:\windows\INF\wnetvsc.inf succeeded
2014-08-13 18:18:55.326, Info      Installing device root\kdnic X:\windows\INF\kdnic.inf succeeded
2014-08-13 18:18:55.420, Info      Spent 735ms installing network drivers
2014-08-13 18:20:25.752, Info      Spent 90234ms confirming network initialization; status 0x003d0002
2014-08-13 18:20:25.752, Info      STATUS: SUCCESS (0x003d0002) = WPEUTIL_S_NET_CONNECTIVITY_VERIFICATION_TIMEOUT

[Scenario #3 wpeinit.log DHCP Disabled on subnet and WinPE Unattend setting DHCP to false]

Total 8 seconds
2014-08-13 18:01:01.893, Info      Spent 1062ms installing network components
2014-08-13 18:01:02.361, Info      Installing device vmbus\{f8615163-df3e-46c5-913f-f2d2f965ed0e} X:\windows\INF\wnetvsc.inf succeeded
2014-08-13 18:01:02.486, Info      Installing device root\kdnic X:\windows\INF\kdnic.inf succeeded
2014-08-13 18:01:02.580, Info      Spent 688ms installing network drivers
2014-08-13 18:01:02.736, Info      UGC process exit code is 0x00000000
2014-08-13 18:01:02.736, Info      Executed UGC; identity[Microsoft-Windows-TCPIP, Culture=neutral, Version=6.3.9600.16384, PublicKeyToken=31bf3856ad364e35, ProcessorArchitecture=amd64, versionScope=NonSxS], command[netiougc.exe -online], status[0x00000000]
2014-08-13 18:01:09.768, Info      QueryAdapterStatus: found operational adapter with static address assigned.

Documentation to set DHCP to false can be found here.  You can find the default unattends for MDT located in the following directory C:\Program Files\Microsoft Deployment Toolkit\Templates.  When you update or rebuild media for deployment point the appropriate unattend will be injected into the PE LiteTouch image. (please backup the originals before making modifications with ADK)

Sample unattend.xml for Windows PE X64 with this configuration:

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="windowsPE">
        <component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State">
            <Display>
                <ColorDepth>32</ColorDepth>
                <HorizontalResolution>1024</HorizontalResolution>
                <RefreshRate>60</RefreshRate>
                <VerticalResolution>768</VerticalResolution>
            </Display>
            <RunSynchronous>
                <RunSynchronousCommand wcm:action="add">
                    <Description>Lite Touch PE</Description>
                    <Order>1</Order>
                    <Path>wscript.exe X:\Deploy\Scripts\LiteTouch.wsf</Path>
                </RunSynchronousCommand>
            </RunSynchronous>
        </component>
        <component name="Microsoft-Windows-TCPIP" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
            <Interfaces>
                <Interface wcm:action="add">
                    <Ipv4Settings>
                        <DhcpEnabled>false</DhcpEnabled>
                    </Ipv4Settings>
                    <Ipv6Settings>
                        <DhcpEnabled>false</DhcpEnabled>
                    </Ipv6Settings>
                    <Identifier>Ethernet</Identifier>
                </Interface>
            </Interfaces>
        </component>
    </settings>
    <cpi:offlineImage cpi:source="wim:c:/temp/delme/x64/install.wim#Windows 7 ENTERPRISE" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

In green I've defined the name of the adapter which is key.  For example, on Hyper-V it was called "Ethernet" but on VMWare "Ethernet0".  If you're unsure, in WinPE you can press F8 to launch a command window and list network information for clarification.

netsh interface ipv4 show interfaces

I thought this was an interesting problem because often the smallest changes which on the surface wouldn't appear to have an impact often do.  So if its 1.5 minutes delay per deployment, does it matter?  If you consider the possibility of thousands of deployments it can add up to quite a bit of time and well worth the investigation.

I received a call today from one of my customers who was experiencing intermittent server hangs and time drift up to five minutes on a mission critical application server. No intervention by the customer was required to bring system back in service, it sorted it out on its own. In the back of my mind, I had already jumped to conclusions about root cause related to Windows 2003 X86 kernel memory pressure exacerbated by deployment of legacy software on modern hardware. I am sure we all have worked those cases and have been able to identify similar problems quickly using tools like poolmon. As a matter of fact, I blogged about a case like that in the past here.

I was wrong. The problem was occurring on Windows 2008 R2 SP1 where previous limitations in kernel memory no longer are relevant since architecture has changed from X86 to X64. Additional information provided by was project used virtual machine deployment on 3^rd party hypervisor only, no dedicated hardware had been deployed. Also, the virtual machine of primary concern was running alongside other virtual machines not related to project, it’s a shared environment and owners of other machines had never reported a problem to date. In many ways I prefer virtual environment because it streamlines number of kernel mode drivers which are typically installed versus hardware scenario.

The hangs and time drift symptoms were happening multiple times a day so we started by collecting baseline performance data using performance monitor. In reviewing the performance data collect during hang timeframe, system did not show any clues such as slow disk I/O or high CPU. However, there was approximately a five minute time window, matching time window as described by customer, where all data points were dropped, simply a gap in time. In Windows 2003 timeframe, there was no native instrumentation for this scenario. Typically we would ask for customer to generate a manual kernel crash dump and provide for review. This is tougher than it sounds from customer logistics standpoint. This would mean around the clock monitoring, adding registry keys to facilitate crash and interruption of business while machine writes contents of RAM to pagefile. It’s important to note, a kernel dump is a representation of a single point of time not a recorder from which we could watch how the system performs over a small period of time.

Enter Windows Performance Recorder which ships with Windows Assessment and Deployment Kit (ADK) for Windows® 8. This toolset leverages ETW which is defined as “general-purpose, high-speed tracing facility provided by the operating system. Using a buffering and logging mechanism implemented in the kernel, ETW provides a tracing mechanism for events raised by both user-mode applications and kernel-mode device drivers.” Where performance monitor collects one sample per second, Xperf will collect thousands per second, therefore large datasets will be collected. Full explanation of ETW and its implementation can be found here. The WPRUI.exe tools makes collecting a trace easy but in this scenario we need a method to collect at least 30 minutes of data needed a circular buffer. This command accomplishes those requirements.

[Xperf start command from elevated command prompt]

Xperf -on Latency+DISPATCHER -stackWalk CSwitch+ReadyThread+ThreadCreate+Profile -BufferSize 1024 -MaxBuffers 1024 -MaxFile 4096 -FileMode Circular

Xperf stop command:

Xperf -d SLOWHANG.etl

Armed with this information, my customer was able to collect traces and made them available for review. Since the window of time for this investigation is quite small, I found it helpful to document the server from which the trace was collected and time trace was collected which is in UTC. In WPA, if you click on “Trace” from the toolbar and then “System Configuration”, we get basic information about system.

The “Traces” item will provide time in UTC.

As a side note, you can run the following command to convert UTC to your time zone.

xperf.exe -i D:\SLOWHANG.etl -a tracestats

Opening the trace, looking at “Generic Events, Activity by Provider, Task,Opcode” you can see time period of exactly 300 seconds where nothing is reported. (5 minute period can be determines by simply left clicking on the for and dragging cursor to right for the period where data points are missing)

Looking at some of the other charts such as “CPU usage” we can clearly see a big drop is activity. In talking with one of our Escalation Engineers, Trey Nash from Global Escalation Services, he made a great point which ultimately led to a break in the case. Can we verify Windows is receiving clock interrupts from the HAL? Let’s take a moment to define what he means referencing Windows Internals book fifth edition.

“The length of the clock interval varies according to the hardware platform. The frequency of the clock interrupts is up to the HAL, not the kernel. For example, the clock interval for most x86 uniprocessors is about 10 milliseconds, and for most x86 and x64 multiprocessors it is about 15 milliseconds. This clock interval is stored in the kernel variable KeMaximumIncrement as hundreds of nanoseconds.” If the clock interrupt is not firing, it will negatively affect the dispatcher code and how it decides when a thread has exceeded its allotted time on the processor, known as quantum accounting. Additionally, essential mechanisms such as system timers will freeze and will not fire on time, or at all during the drop-out window, thus leading to apparent hang situations since many mechanisms within the OS rely on timers.

So let’s verify clock interrupts for the entire period and baseline number of them received for 1 minute period when things are “working”.

[Using DPC/ISR DPC/ISR Duration by Module, Function we can use handy “filter to selection” to focus on clock interrupts]

(Function under hal.dll is HalpHpetClockInterrupt)

Wow, we can see clearly that during the reported problem timeframe a gap of exactly 300 seconds no clock interrupts were received. How many should we expect to see during 1 second? After zooming into working period of roughly one second, you can locate the count by clicking on the “display graph and table” item on top right.

So for roughly one second we receive about ~1,500 clock interrupts.

So we should expect 450,000 or so for 300 seconds. How many did we receive during problem time period? Zero.

So not receiving clock interrupts from HAL is our root cause. If this was on hardware, this could potentially be bad processor etc. Since this is on hypervisor, we need to understand why that is happening. In this case, 3^rd party vendor worked around issue by disabling HPET on a per VM basis and is working to develop a patch to address issue. In troubleshooting this case, we were able to collect many traces, in every instance where hang occurred, no interrupts were received during problem timeframe. As it turns out, the customer was experiencing this issue on multiple VM’s but were not aware of it because they were only tightly monitoring this one VM which was running critical workload.

I found this problem scenario and case to really highlight power and usefulness of ETW tracing. These tools and methods made a big difference in identifying root cause and stabilizing customer environment and perhaps will help you in your troubleshooting efforts.

The customer environment was fairly straightforward, each Hyper-V host was a domain member and for testing purposes the physical machines themselves were actually located in the same datacenter with gigabit connectivity.  Since this feature was new to me, I started out by first setting it up from scratch in a test environment using commodity hardware which consisted of a DC and two member servers. Armed with this new knowledge and walking through deployment guide above I felt more confident continuing in the investigation.

Domain - REPLICADOMAIN

DC - REPLICADC

Hyper-V Hosts - REPLICAHOST1 (Primary), REPLICAHOST2 (Replica)

 In my customer environment, any attempt to enable replication for a VM returned the following error, 100% repro.  Replica uses mutual authentication based on Kerberos or mutual authentication based on certificates for the server (not for the user).

First place to start is reviewing Windows event logs for items of interest.  I reviewed the following logs on each machine:

System, Application, Security and Microsoft-Windows-Hyper-V-VMMS.  In the latter, I found:

02/04/2013 01:12:06 PM   Error         ITSUSRALAB039.df 32000   Microsoft-Windows-Hyper-V-VMMS      N/A                NT AUTHORITY\SYSTEM               Hyper-V failed to enable replication for virtual machine 'YOURVMNAME': The connection with the server was terminated abnormally (0x00002EFE).

02/01/2013 03:35:17 PM   Error         ITSUSRALAB039.df 29212   Microsoft-Windows-Hyper-V-VMMS      N/A                NT AUTHORITY\SYSTEM               Hyper-V failed to authenticate the primary server using Kerberos authentication. Error: The logon attempt failed (0x8009030C)

You can use the Microsoft Exchange Server Error Code Look-up tool called ERR.exe to help convert the error codes to text.

err 0x00002EFE
# for hex 0x2efe / decimal 12030
  ERROR_INTERNET_CONNECTION_ABORTED                              inetmsg.h
  ERROR_WINHTTP_CONNECTION_ERROR                                    winhttp.h
  ERROR_INTERNET_CONNECTION_ABORTED                              wininet.h
# 3 matches found for "0x00002EFE"

err 0x8009030C
# for hex 0x8009030c / decimal -2146893044
  SEC_E_LOGON_DENIED                                                                  winerror.h

The second one was a great tip but unfortunately I didn't put it all together.....  so we did more troubleshooting.

The deployment guide provides specific guidance on what is required for Kerberos delegation and this seemed like a reasonable place to start as well as verifying that all the SPNs are registered.  There are a couple ways to verify this information.  One would be to use ldifde.exe to dump all the configuration settings of the computer object in Active Directory.  The command using ldifde.exe would this:

[Each command would need to be executed against each Hyper-V hosts computer account]

C:\>ldifde -d "CN=REPLICAHOST1,CN=Computers,DC=REPLICADOMAIN,DC=com" -f REPLICAHOST1.log
Connecting to "REPLICADC.REPLICADOMAIN.com"
Logging in as current user using SSPI
Exporting directory to file output.log
Searching for entries...
Writing out entries..
2 entries exported

The command has completed successfully

To retrieve the SPN's for each computer object we can leverage setspn.exe, the command would be:

C:\>setspn -L REPLICADOMAIN\REPLICAHOST1

Registered ServicePrincipalNames for CN=REPLICAHOST1,CN=Computers,DC=REPLICADOMAIN,DC=com:
 
 Hyper-V Replica Service/REPLICAHOST1
 Hyper-V Replica Service/REPLICAHOST1.REPLICADOMAIN.com
 Microsoft Virtual System Migration Service/REPLICAHOST1
 Microsoft Virtual System Migration Service/REPLICAHOST1.REPLICADOMAIN.com
 Microsoft Virtual Console Service/REPLICAHOST1
 Microsoft Virtual Console Service/REPLICAHOST1.REPLICADOMAIN.com
 WSMAN/REPLICAHOST1
 WSMAN/REPLICAHOST1.REPLICADOMAIN.com
 TERMSRV/REPLICAHOST1
 TERMSRV/REPLICAHOST1.REPLICADOMAIN.com
 HOST/REPLICAHOST1
 HOST/REPLICAHOST1.REPLICADOMAIN.com

Next we need to verify that the computer objects have Kerberos delegated correctly:

 Verify each Hyper-V host can successfully communicate with Active Directory:

C:\>nltest /SC_VERIFY:REPLICADOMAIN
Flags: b0 HAS_IP  HAS_TIMESERV
Trusted DC Name \\REPLICADC.REPLICADOMAIN.com
Trusted DC Connection Status Status = 0 0x0 NERR_Success
Trust Verification Status = 0 0x0 NERR_Success
The command completed successfully

 Verify that KDC is assigning Kerberos tickets:

C:>klist

Current LogonId is 0:0x3364e

Cached Tickets: (7)

#0> Client: Administrator @ REPLICADOMAIN.COM
 Server: krbtgt/REPLICADOMAIN.COM @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x60a10000 -> forwardable forwarded renewable pre_authent name_canonicalize
 Start Time: 2/6/2013 15:38:20 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0x2 -> DELEGATION
 Kdc Called: REPLICADC.REPLICADOMAIN.com

#1> Client: Administrator @ REPLICADOMAIN.COM
 Server: krbtgt/REPLICADOMAIN.COM @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
 Start Time: 2/6/2013 14:36:21 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0x1 -> PRIMARY
 Kdc Called: REPLICADC.REPLICADOMAIN.com

#2> Client: Administrator @ REPLICADOMAIN.COM
 Server: cifs/REPLICADC.REPLICADOMAIN.com @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize
 Start Time: 2/6/2013 15:38:20 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0
 Kdc Called: REPLICADC.REPLICADOMAIN.com

#3> Client: Administrator @ REPLICADOMAIN.COM
 Server: ldap/REPLICADC.REPLICADOMAIN.com @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize
 Start Time: 2/6/2013 15:38:20 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0
 Kdc Called: REPLICADC.REPLICADOMAIN.com

#4> Client: Administrator @ REPLICADOMAIN.COM
 Server: RPCSS/REPLICAHOST2 @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
 Start Time: 2/6/2013 14:39:01 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0
 Kdc Called: REPLICADC.REPLICADOMAIN.com

#5> Client: Administrator @ REPLICADOMAIN.COM
 Server: LDAP/REPLICADC.REPLICADOMAIN.com/REPLICADOMAIN.com @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize
 Start Time: 2/6/2013 14:36:53 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0
 Kdc Called: REPLICADC.REPLICADOMAIN.com

#6> Client: Administrator @ REPLICADOMAIN.COM
 Server: RPCSS/REPLICAHOST2.REPLICADOMAIN.com @ REPLICADOMAIN.COM
 KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
 Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
 Start Time: 2/6/2013 14:36:21 (local)
 End Time:   2/7/2013 0:36:21 (local)
 Renew Time: 2/13/2013 14:36:21 (local)
 Session Key Type: AES-256-CTS-HMAC-SHA1-96
 Cache Flags: 0
 Kdc Called: REPLICADC.REPLICADOMAIN.com

So far everything looks good, no indications of a Kerberos problem (darn it!!).  At this point, we moved on to running a network trace from Primary and Replica server simultaneously. In previous versions of Windows, we would need to install Netmon or Wireshark to collect this information.  A really great trick is leveraging ETW providers to collect the trace.

[Commands to start trace]

ipconfig /flushdns
klist purge
netsh trace start capture = yes

*repro issue*

[Commands to stop trace]

netsh trace stop

You can then copy over the ETL trace files to your local machine and view with Netmon.  Unfortunately, the network trace revealed no clues on where to look next. 

When in doubt, always turn to Process Monitor  and collect a trace on both sides, I didn't find any Access_Denied or anything else of interest. 

At this point, we decided to refocus on the error itself, Kerberos.  I enabled Kerberos debug logging on each Hyper-V machine, reproduced the issue and then stopped trace. Here is the command:

[Start the trace]
Logman.exe start kerb -p "Security: Kerberos Authentication" 0x40043 -o .\kerb.etl -ets

Run through the Hyper-V wizard and stop traces immediately when exception is raised.

[Stop the tace]
Logman.exe stop kerb –ets

Rename Kerb.etl to include the respective server name from both servers.  Unfortunately, in order to convert this trace you will need to engage CTS Support.  My trace revealed a clue which really helped:

Failed to create token: 0xc000015b.  Let's convert that:

err.exe 0xc000015b

STATUS_LOGON_TYPE_NOT_GRANTED

To me this has more to do with user rights security rather than Kerberos security.  In comparing user right assignments between two environments, I noticed a difference which would lead to a break in the case.  From the Local Group Policy Editor, here are the differences:

[Working - Windows Default]

[Broken - Customer environment]

Now we are getting somewhere!  So if this is a security issue, shouldn't an event have been written to Security log which I reviewed in the beginning?  Sigh... Yes.  I simple missed it because I only looked at the Security Log on the Primary machine not the Replica.  Going back to the Security Log I found the following event on the Replica Server:

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          2/6/2013 12:02:01 PM
Event ID:      4625
Task Category: Logon
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      REPLICAHOST2.REPLICADOMAIN.com
Description:
An account failed to log on.

Subject:
 Security ID:  NULL SID
 Account Name:  -
 Account Domain:  -
 Logon ID:  0x0

Logon Type:   3

Account For Which Logon Failed:
 Security ID:  NULL SID
 Account Name:  REPLICAHOST1$
 Account Domain:  REPLICADOMAIN.COM

Failure Information:
 Failure Reason:  The user has not been granted the requested logon type at this machine.
 Status:   0xC000015B
 Sub Status:  0x0

Process Information:
 Caller Process ID: 0x0
 Caller Process Name: -

Network Information:
 Workstation Name: -
 Source Network Address: -
 Source Port:  -

Detailed Authentication Information:
 Logon Process:  Kerberos
 Authentication Package: Kerberos
 Transited Services: -
 Package Name (NTLM only): -
 Key Length:  0

This particular customer has a requirement to harden Windows security where possible so instead of simply adding back in all the Windows defaults we granted "Authenticated Users" group the following user right "Access this computer from the network." to both machines which resolved issue!

Summary:

This was a great reminder for me to always take the necessary time to review all of the event logs before engaging in advanced troubleshooting.  I hope some of the troubleshooting steps we used may be helpful for your future investigations.

Notice my statement "highly available", I didn't say fault tolerant.  You are still getting 99% uptime, but the reality is...your client had an outage.  When the group moves the file share, depending on the application, customer may experience an outage.  The cluster service, in order to provide high availability, routinely checks for the health of the file share to ensure online status.  How does it do that?  It routinely performs a directory listing of the file share itself to ensure it's alive and well.  There are many reasons why this process may fail, not limited to memory pressure, server service load, file or system filter drivers such as A/V, list goes on and on.  Typical errors may look like the following:

Error Messages: ClusSvc Event ID:1055
Cluster File Share resource ‘demo’ has failed a status check. The error code is 64.

00000888.00001108::2009/08/04-02:16:09.021 INFO File Share <DEMO>: Retrying FindFirstFile on error 64 for share \\HAFileShare\demo\*.* !

ERR 64 = The specified network name is no longer available.

The question in my mind is “Does Windows Server 2012 offer a solution to this problem?”  The answer is obvious; otherwise I wouldn’t have written this blog ;)

One of the primary design goals the product team had for Windows Server 2012 was to enable workloads such as SQL Server or Hyper-V to use SMB File Server back end storage as primary storage for datacenter scenarios.  So what does SQL Server and Hyper-V have to do with the scenario above?  Well, part of the framework required for their objective included a major overhaul of SMB to version 3 and to include “transparent failover”.  Windows Server 2012 offers two file server types, the one we all know and love now called “Classic” file server and "scale-out" file server to support workloads above.  The “classic” file server role provides the ability to failover a file share resource with zero downtime and is the scenario we have been discussing. Scale-out file server is a different discussion for another blog.

I know what you’re thinking…this is awesome, all I have to do is migrate by backend file server to Windows Server 2012 and I’m golden.  After all, you’re more than half way complete with Windows XP refresh project to Windows 7. Well… No.  This technology, at its core, depends on SMB3 which is included only in Windows Server 2012 and Windows 8.  Therefore, you must be running those SKU’s to take advantage of this feature. 

For me, seeing is believing, so let’s move from the academic discussion into real world application and testing.  My lab consists of five machines:

Windows Domain controller (iSCSI target for lab scenario)
Two Node Windows Server 2012 Cluster with shared storage using iSCSI and “Classic File Share” configured with continuous availability.
One XP Client
One Windows 8 Client

Test Scenario:
Create continuous available file share and initiate large file transfer from client.  During copying process, move resource group containing file share to other node and note any disruptions.  We know on XP client, any disruption in availability of file share will immediately return error to user.  How will Windows 8 fare?

[Add file share role to each cluster node]

[Select file server for general use, scale-out is for Hyper-V or SQL only]

[Create client access point, the virtual name your clients will use to connect]

[Select a disk resource to host file share]

[Complete wizard]

[HAFileShare is my CAP and it’s online owned by Node 1]

[Create file share for continuous availability]

[Make a selection, notice the other settings item pending]

 
[Define share name]

 
[Default option foe continuous availability is selected]

[Demo Share is online]

[Initiate large file transfer from client to file server on XP, move cluster group containing file share during copy process]

[Network trace snippet] 
1832 1:37:42 PM 10/21/2012 17.3213007 System HAFILESHARE  SMB SMB:C; Nt Create Andx, FileName = \largefile.iso {SMB:639, SMBOverTCP:19, TCP:18, IPv4:17}
245 1:37:30 PM 10/21/2012 5.6650507 System HAFILESHARE   SMB SMB:R; Nt Create Andx - NT Status: System - Error, Code = (52) STATUS_OBJECT_NAME_NOT_FOUND {SMB:30, SMBOverTCP:19, TCP:18, IPv4:17}
287 1:37:30 PM 10/21/2012 5.6650507 System HAFILESHARE SMB SMB:R; Nt Create Andx - NT Status: System - Error, Code = (172) STATUS_PIPE_NOT_AVAILABLE {SMB:39, SMBOverTCP:19, TCP:18, IPv4:17}

[Initiate large file transfer from client to file server on Windows 8, move cluster group containing file share during copy process.  Notice the temporary dip in transfer rate but no exception, copy completes]

[Event logs on the Windows 8 machine] 

Log Name:      Microsoft-Windows-SMBClient/Operational
Source:        Microsoft-Windows-SMBClient
Event ID:      30624
Level:         Information
User:          SYSTEM
Description:
Connection to share \hafileshare\demo was re-established.

Log Name:      Microsoft-Windows-SMBClient/Operational
Source:        Microsoft-Windows-SMBClient
Event ID:      30623
Level:         Warning
Description:
Connection to share \hafileshare\demo was lost. Status 0xC000020C

Summary:
Continuous available file share is a great new feature and I hope you will evaluate Windows Server 2012 and Windows 8 with the common goal to provide your business partner with highly available and fault tolerant services.
 

In the past for their business, each application required a single application partition.  As a result, their standard procedure included provisioning a new ADAM instance with a single application partition.  The problem started when the requirements slightly changed.  The new line of business application required two application partitions not one.  They used their existing procedures for setting up one application partition for two application partitions scenario. Lab certification was failing due to intermittant errors in lab with error "0000202B: RefErr: DSID-03100697 A referral was returned from the server." when attempting to write to second application partition.  They could not recreate the error when writing to first partition in a consistent manner.

For clarity let's walk through lab setup.  Let's assume for the moment we have a domain called ACME.COM and our ADAM servers are called ADAM1.ACME.COM and ADAM2.ACME.COM.  Let's create our ADAM instance and call it something simple and straight foward like "BlogInstance".  Additionally, we can call our application partitions "BlogPartition1" and "BlogPartition2".  Here are a few screen shots for initial setup to create the partition and first partition from which we can leverage the ADAM wizard.

 [Use wizard to create ADAM instance]

[Specify ports]

[Create and name the application partition, first one of course]

[Import in box ldif files to create classes etc.]

OK.  For brevity, I'll skip the rest of the wizards, next button throughout.  So now in services on ADAM1.ACME.COM we can verify that new NT Service is installed.

As I mentioned, process is very important as it should be to the customer.  Next step is to add ADAM2.ACME.COM to replica.  From ADAM2.ACME.COM, let's run the ADAM wizard and create replica of existing instance.

[From ADAM2]

[Notice the application partitions available to copy, we have only created one and therefore this is only one presented]

At this point, the wizard completes successfully and we verify replication between ADAM1.ACME.COM and ADAM2.ACME.COM is working using command such as C:\windows\adam>repadmin /syncall ADAM1:389.  So far so good.  Next step which is new territory includes creation of second application partition for instance "BlogInstance".  Let's say the second application partition is called "BlogPartition2" and is created on ADAM1.ACME.COM.

If you need to create the second partition for the application you can use dsmgmt.exe to accomplish task.

[Create second partition called BlogPartition2]

So now we have the same setup as the customer.  Sure enough, if I try to create an object in the second partition from ADAM2.ACME.COM I get a referral error.  Being able to reproduce the problem is huge in finding a resolution.  Through research and collaboration with teammates an attribute, msDS-NC-Replica-Locations, was identified which could explain issue. This attribute was populated with both ADAM1.ACME.COM and ADAM2.ACME.COM for first partition where no problems were encountered.  However, this same attribute for the second partition only listed one server where second partition was originally created, in our case ADAM1.ACME.COM.  I think we are on to something here as our referral error goes away when executed from server that is populated in attribute msDS-NC-Replica-Locations.We need to understand how and why this value gets populated. 

From MSDN:

2.194 Attribute msDS-NC-Replica-Locations

This attribute specifies a list of servers that are the replica set for the corresponding non-domain naming context.

cn: ms-DS-NC-Replica-Locations
ldapDisplayName: msDS-NC-Replica-Locations
attributeId: 1.2.840.113556.1.4.1661
attributeSyntax: 2.5.5.1
omSyntax: 127
omObjectClass: 1.3.12.2.1011.28.0.714
isSingleValued: FALSE
schemaIdGuid: 97de9615-b537-46bc-ac0f-10720f3909f3
systemOnly: FALSE
searchFlags: 0
linkID: 1044
systemFlags: FLAG_SCHEMA_BASE_OBJECT

[Partition 1, notice partition 1 lists both ADAM1.ACME.COM and ADAM2.ACME.COM]

[Partition 2 only contains one entry for ADAM1.ACME.COM]

How does referral error factor into this?  The application making the query is reponsible for "chasing" a referral.  In our case, ldifde.exe does not chase referrals, it's simply saying in order to make a change to second application partition, the change is required to be made on the server defined by msDS-NC-Replica-Locations attribute.

So how can we resolve the issue, customer preference is command line so setup can be automated.

1. C:\windows\ADAM>dsmgmt.exe
2. partition management
3. Connections
4. server connections: connect to server ADAM1:389  (Where ADAM1 is server where second partition was created and port is 389)
5. server connections: q
6. partition management: add NC Replica "DC=BlogPartition2,DC=ACME,DC=COM" ADAM2:389      (Where BlogPartition2 is second partition and you want to add ADAM2 to host NC replica)
8. partition management: q
8. dsmgmt: q

Let's verify that second application partition msDS-NC-Replica-Locations attribute includes both servers.  Simply go back to ADSI-Edit and check second partition, it will now show both servers.  Time to see if we can create object in second partition from ADAM2.ACME.COM as previously we received referral error.

Using LDIFDE.exe, placing ldf file in root of C:\ for simplicity.

[LDF - CreateContainer.ldf]

dn: CN=CreatOnADAM2,CN=Roles,DC=BlogPartition2,DC=ACME,DC=COM
changetype: add
objectClass: container

[ACME2.ACME.COM]

Success!

I hope this is helpful, I certainly learned quite a bit about ADAM\AD LDS and specifically purpose of Attribute msDS-NC-Replica-Locations as well as "Add NC Replica"

The first question of course is the infamous “What has changed?”  Fortunately, customer had a formal change process and could detail changes made before problem.  In a nutshell:

• Recommended hotfixes and updates for Windows Server 2008 R2 SP1 Failover Clusters
  https://support.microsoft.com/kb/2545685
• The network location profile changes from "Domain" to "Public" in Windows 7 or in Windows Server 2008 R2
  https://support.microsoft.com/kb/2524478 
• Last month’s security updates via Windows Update 
• Latest HP Proliant Support Pack

At the time of the problem I had the following thoughts:

• Can’t be the NetAPP appliance because it can be accessed from other Windows machine outside of Cluster
• Can’t be the Hyper-V host because from each Hyper-V host, the parent partition, the Administrator could access the share
• Not a hardware problem since the issue is present on both cluster nodes
• Highly unlikely Windows security patches are relevant especially since they were applied only to the Hyper-V hosts not the virtual machines
• Highly unlikely recommended updates to cluster as they will be included in SP2 for 2008 R2
• Can’t be security because the same Administrator who accesses the share from Hyper-V host gets the permission error within the VM, therefore the dialog box is bogus

The problem with train of thought is I essentially ruled out everything….  We tried all the usual steps to collect more information about problem such as connecting/reconnecting virtual NICs within VM’s, changed from synthetic to legacy adapters within VM’s, restarted VM’s, restarted Hyper-V hosts etc. etc. 

After a lot of testing through trial and error, a suggestion was made to roll back the network card driver which was updated in the HP PSP to latest version on one of the cluster nodes.  Immediately after replacing the latest driver with older driver problem was solved.  Evidently a bug exists in new driver which prevented only the virtual machines from accessing the NetAPP appliance, truly a strange one indeed.

The problematic driver was:

In summary, with multiple layers of virtualization and integration with 3rd party devices, troubleshooting and identifying where the problem resides can certainly be difficult.  One lesson I learned which may have made identification of the problem easier would have been updating only one cluster node at a time and having client certify and sign off before patching second node.  Of course, I am not sure that a basic certification would have even revealed problem unless someone attempted to access NetAPP device from within VM.

What Operating System and Service Pack? Windows 2003 Enterprise SP2 X86

Was RDP possible during unresponsiveness? No

Does server return to normal operation after some amount of time? No

Does the server respond to ping? Yes

Do they have Poolmon data for review? Yes

Did they manually capture a crash dump that we could review? No. Suggested to customer Windows feature lets you generate a memory dump file by using the keyboard

Did they collect MPS Reports for review? Yes. Microsoft Product Support Reports 

At this point I have some data to review and giving the customer a hard time about deploying an operating system in extended support with 32 bit architecture was unlikely to be well received. 

[System Event Log - beginning with clean start of system]

System 7/14/2011
1:04  Information        6009       EventLog                             POOLMONEXAMPLE                      Microsoft (R) Windows (R) 5.02. 3790 Service Pack 2 Multiprocessor Free.

System 7/14/2011
8:29  Error                  2019       Srv                                       POOLMONEXAMPLE                      The server was unable to allocate from the system nonpaged pool because the pool was empty.

So, luckily enough, we have something interesting only seven hours and twenty-five minutes later and after clean boot, kernel nonpaged pool appears to be depleted?  Why? We have a time frame that we should be able to correlate with Poolmon logs.  You may already be familiar with Poolmon but by definition the tool is designed to “display data that the operating system collects about memory allocations from the system paged and nonpaged kernel pools and about the memory pool”.  Microsoft CTS Support has a very simple script to put the collection in place if the KB by itself is not sufficient. 

Ok, so on to Poolmon, let’s start with when the system was clean booted for baseline information. 

[Clean boot happened at 7/14/2011 1:04]

First sample from Poolmon is shown below.  The N: stands for nonpaged pool and P: stands for Paged Pool.  The error above stated that the system ran out of nonpaged pool so we may want to keep a close eye on that number which is currently 26 MB.  It’s important to note that 32 bit architecture is severely limited by the amount of paged and nonpaged pool accessible at startup. This is not a Windows limitation, per se, but a mathematical one.  Having said that, what is the maximum amount of nonpaged pool which Windows can use on this platform and SKU? Roughly 256MB. Our friend and Windows expert Mark Russinovich has written a definitive article on this subject: Pushing the Limits of Windows: Paged and Nonpaged Pool

Next sample shows that nonpaged pool has grown at an alarming rate.

Now we are getting somewhere.  We know what is happening, next step is to determine root cause by identifying offending driver.  At first glance this appears too difficult to read, but for what we are after its relatively straight forward.  Thankfully, poolmon sorts the drivers based on tags and the amount of pages consumed. When drivers in kernel space need memory, they ask the memory manager to assign a page of memory.  There is nothing wrong for systems under load to take memory, but it is important to free that memory when load diminishes.  Some of the key fields are: Memory/Available memory – no problem there, Pool N or nonpaged is maxed out which is our problem, paged pool is fine. Looking at the tags listed, you see a tag/driver called “Tidt”which has consumed 214 MB which is almost all available nonpaged pool where the next highest consuming driver only used 35 MB.  We need to know what developer owns TidT tag?  A few searches on the internet, unfortunately, did not yield any promising clues.  A teammate shared a trick where you can parse all the files in system32 directory to locate a matching tag name. It is common practice that tag names may be in the header of the image file.  This is the command I used:

C:\WINDOWS\system32\drivers>findstr TidT *.*
LhmonProxy.sys:...LhmonIoCompletion...c:\winstaps\v81\tap\win\src\lhmonproxy\ioc
ompletion.c...LhmonProxy.sys..................ï Uï.â.?í$Ç..Sï]?Vïp(3.ëE.ëE°èC!ä.
Wëunt.ïC`ÇH..SF[.  SF.²  SF_.  ï}.à .ä....ìN. ..p..ï.ïw.ë.ëq.ïMnèQ.ä.t.ïw.à÷ïW?ë
U?t. A?d.ïu.è.â.. ..p..à÷.ä....èC$ä.t.èG.ä.u.ïC.à.|?èO.ä.u?à.}_èG.ä.tXSh.O..j.hÆ
...h.P..h(P..h`P..FU...ïE?ïM.â..PSQ .ShpP..j.hö...höP..h¿P..haP..ëE°F#...â..d.ïE
?à.t    j.P .Lp..ïunâ..ï. ..p..ïMn I?è.ï. ..p..WFl...ïE°_^[ïs].?.ïG?à.t j.P .Lp.
.WFJ...ïE°_^[ïs].?...............ï Uï.Vïu?èF!ä.t.ïF`ÇH..VF..  VFìn  VF.n  3.^].?
.................ï Uï.â..í$Ç..SVïp(ì^.Wï.ë]° ..p..ì~.ï7;.êE t!3.ÉïF?;.ë^.t?SP .L
p..ë^?ï6;.usï]°èU ï. ..p.._^[ïs].................ï Uï.SVWï°í$Ç..ïX(F9...ï.à÷tZïM
.ïU?èE.ëN.èM.ëV?èU.êN.êF.êV..G..Q..ëw .G.aì{.ï. ..p..ìK.è.ïA.ë.ëF.ë0ëq.ï. ..p.._
^3.[]..._^.....[]................ï Uï.ïM.ïU.ïE.ï@`QïM.RïU.QïM?RQFL   ]..........
ïM.PëH? .tp..]........ï Uï.QVï. ..p..ìw.êE 96t.Sï.Lp..ï.ï.j.ë.Pëq. .96ue[èU 3.ï.
ëG?ëG.ëG.ëG. ..p..^ïs]................ï SWï. ..p..ïN.è.ìF.;.t.ï.ë.ëB.ïF?ï.Hè.ï.ë
F? ..p..ï._[.ïF.ïN.PQj. ..p..è.ï.ï° ..p..ï._[.........ï Vï5(p..Wh`í.. ..dí..3 h@
í..úhí..údí..ë=lí....tí..(...ë=pí....xí..xtCE ..Dí..haá..úHí..úDí..ë=Lí....Tí...
...ë=Pí....Xí..gToI ..Sá..h.í..úFá..úSá..ë=.á.....á..X...ë=.á....°á..TidT ...í..
h í..ú.í..ú.í..ë=?í.....í..P...ë=.í.....í..CidT ..$í..h.á..ú(í..ú$í..ë=,í....4í.
.....ë=0í....8í..gTcA ...á..ë=.á..ë=.á.._ú.á..ú.á.....á..?......á..gTcD^........
......ï W.`í..F.²  .@í..F.²  .aá..F.²  ..í..F.²  . í..F½²  ..á..Fí²  _..........
......ï V.`í..Fp²  à.^t!3.ï.ë

This points us to the lhmonproxy.sys driver written by 3^rd party vendor.  The customer, armed with this information, was able to obtain a patch from the vendor to resolve the problem.  I hope this blog is helpful providing a real world example of troubleshooting with poolmon and provides further encouragement to move towards X64 platform.