Hi, my name is Deepak and today I will be talking about a common scenario with IE11 upgrade from earlier versions fails with CRYPT_E_NOT_FOUND.

To fix this, we need to cleanup the existing IE11 installation. If you browse to C:\windows\servicing\packages, you will notice that there are a lot of files for IE11 and using each one of them to uninstall would be a tedious process. There is, however, a way to remove them in one go. The below command assists us in doing that.

FORFILES /P %WINDIR%\servicing\Packages /M Microsoft-Windows-InternetExplorer-*11.*.mum /c "cmd /c echo Uninstalling package @fname && start /w pkgmgr /up:@fname /quiet /norestart"

What is does is that it passes each IE 11 MUM file as an argument to pkgmgr which further process it to uninstall the package. Please note that this is applicable only for Windows Server 2008 R2/ Windows 7, Windows Server 2008.

NOTE : Please note that the machine needs to be rebooted post running the command.

If the MUM and CAT files for IE11 still exist in C:\windows\servicing\packages , they would have to be manually removed.

Deepak Gopaluni

Senior Support Engineer |Microsoft Windows Core

Disclaimer : This information is provided ‘as-is’ with no warranties

Hi, my name is Prachi and today I will be talking about a patch installation problem which occurs when multiple patches are installed together.

Consider the following scenario

You have a scheduled task\script that scans for updates and while it detects multiple updates, it fails when installing them. If you check the update history, you see that it failed to install any update after it tried to install a security update. The security update, then shows reboot required. 

When you open the CBS logs located in C:\windows\logs\CBS, you see the following:

2015-08-30 04:25:29, Info                  CSI    00000235 Component change list:
2015-08-30 04:25:29, Info                  CSI    00000236 Couldn't find the hash of component: Microsoft-Windows-BootEnvironment-OS-Loader.Resources, Version = 6.1.7601.23136, pA = PROCESSOR_ARCHITECTURE_AMD64 (9), Culture = [l:10{5}]"en-US", VersionScope = 1 nonSxS, PublicKeyToken = {l:8 b:31bf3856ad364e35}, Type neutral, TypeName neutral, PublicKey neutral in the catalog [l:192{96}]"\SystemRoot\WinSxS\Catalogs\33aaa08f767a1d074889ca79e8ed861442eef084bcbaaa5379376ad63e2c87d5.cat".
2015-08-30 04:25:29, Error                 CSI    00000237@2015/8/30:04:25:29.184 (F) : Error 80092004 [Warning,Facility=FACILITY_NTSSPI,Code=8196 (0x2004)] originated in function CCSDirectTransaction::AddImplicationsToCatalogsAndVerifyComponentHashes expression: (null)
[gle=0x80004005]
2015-08-30 04:26:07, Error                 CSI    00000238 (F) 80092004 [Warning,Facility=FACILITY_NTSSPI,Code=8196 (0x2004)] #14706236# from CCSDirectTransaction::PrepareForCommit(...)[gle=0x90092004]
2015-08-30 04:26:07, Error                 CSI    00000239 (F) 80092004 [Warning,Facility=FACILITY_NTSSPI,Code=8196 (0x2004)] #14706235# from CCSDirectTransaction::ExamineTransaction(...)[gle=0x90092004]
2015-08-30 04:26:07, Error                 CSI    0000023a (F) 80092004 [Warning,Facility=FACILITY_NTSSPI,Code=8196 (0x2004)] #14706234# from CCSDirectTransaction_IRtlTransaction::ExamineTransaction(...)[gle=0x90092004]
2015-08-30 04:26:07, Error                 CSI    0000023b (F) CRYPT_E_NOT_FOUND #14705449# from Windows::COM::CPendingTransaction::IStorePendingTransaction_Analyze(...)[gle=0x80092004]
2015-08-30 04:26:07, Error                 CSI    0000023c (F) CRYPT_E_NOT_FOUND #14534227# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_Commit(Flags = 47 (0x0000002f), pSink = NULL, disp = 0, coldpatching = FALSE)[gle=0x80092004]
2015-08-30 04:26:07, Error                 CSI    0000023d (F) CRYPT_E_NOT_FOUND #14534226# 42410326 us from Windows::ServicingAPI::CCSITransaction_ICSITransaction::Commit(flags = 0x0000002f, pSink = NULL, disp = 0)
[gle=0x80092004]
2015-08-30 04:26:07, Error                 CBS    Exec: Failed to commit CSI transaction to resolve execution chain. [HRESULT = 0x80092004 - CRYPT_E_NOT_FOUND]
2015-08-30 04:26:07, Info                  CSI    0000023e@2015/8/30:04:26:07.856 CSI Transaction @0x2feaf1b0 destroyed
2015-08-30 04:26:07, Info                  CBS    Perf: Resolve chain complete.
2015-08-30 04:26:07, Info                  CBS    Failed to resolve execution chain. [HRESULT = 0x80092004 - CRYPT_E_NOT_FOUND]
2015-08-30 04:26:07, Error                 CBS    Failed to process single phase execution. [HRESULT = 0x80092004 - CRYPT_E_NOT_FOUND]
2015-08-30 04:26:07, Info                  CBS    WER: Generating failure report for package: Package_for_KB3071756~31bf3856ad364e35~amd64~~6.1.1.2, status: 0x80092004, failure source: Resolve, start state: Absent, target state: Installed, client id: WindowsUpdateAgent

This happens because security updates usually require a reboot to commit and will fail any update that is installed after that. Please note that once you encounter a security update, it always does ask for a reboot. In addition to committing the transactions, a reboot is also responsible for committing a task called scavenging. Please note that scavenging is invoked on system startup (when TrustedInstaller starts on boot), system shutdown and idle processing.

There is another type of reboot called a non-mandatory reboot which only requires scavenging.

You can manually invoke scavenging by stopping and starting the Windows Modules Installer (TrustedInstaller) service. Please note that scavenging, as the name suggests, removes the unused components and it might help with the problem but only when the reboot required is non-mandatory.

If a third party software is being used for this, the following steps can be added to the schedule:

1. Check the patch to see if it is a security patch.
2. If it is a security patch, then install it and restart the Windows Modules Installer (TrustedInstaller) service.
3. If not, then simply install it.

NOTE: Types of RebootRequired in the Windows Update system

There are two levels of “reboot required” within the Windows Update system. There’s “mandatory reboot required”, which basically means “nothing more will happen in Windows Update until you reboot”; and there’s “non-mandatory reboot required”, which basically means “You can’t do new scans, but you can continue to install updates that you already located through previous scans”.

A non-mandatory reboot creates the key:

SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired

A mandatory reboot creates the key

 SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired\Mandatory

Prachi Gupta

Support Engineer |Microsoft Windows Core

Disclaimer : This information is provided ‘as-is’ with no warranties

Hi, my name is Anirudh and today I will be talking about a scenario where the OS deployment using MDT fails with the error ‘Network driver not found’. This issue is not common for the reason that most of the NIC cards work fine with the in-box drivers in Windows PE but more often than not, there would be a situation where this error pops up.

Since we know that this error came after PXE and when we boot into WindowsPE, then we know for sure that the NIC card is working fine.

The first step to follow is to ensure that the NIC drivers are added to your WindowsPE. We should also ensure that for x86 and\or x64 boot image, we have the network drivers included which can be verified by checking the setting as below:

Please note that merely selecting this option is not enough. The WindowsPE WIMs should have these drivers. If someone is not sure whether the drivers are there in WindowsPE, then there is no harm in refreshing the deployment share to create new LiteTouchPE WIMs.

The next step is to look at the setupapi.dev logs which track driver installation failures. The setupapi.dev.log for WindowsPE is located in the folder X:\Windows\inf.

If you track the last entry in setupapi.dev.log, you might see something similar to the following

     dvi:      Class GUID of device changed to: {4d36e972-e325-11ce-bfc1-08002be10318}.
     ndv:      {Core Device Install} 3:41:20.204
     dvi:           {DIF_ALLOW_INSTALL} 3:41:20.329
     dvi:                Using exported function 'NetClassInstaller' in module 'X:\WINDOWS\system32\NetCfgx.dll'.
     dvi:                Class installer == NetCfgx.dll,NetClassInstaller
!!!  dvi:                Loading module 'X:\WINDOWS\system32\wlaninst.dll' failed.
!!!  dvi:                Error 126: The specified module could not be found.
!!!  dvi:                Error 126 loading CoInstaller(wlaninst.dll,WlanDeviceClassCoInstaller)
!!!  dvi:                Error 126: The specified module could not be found.
!!!  dvi:                Loading module 'X:\WINDOWS\system32\wwaninst.dll' failed.
!!!  dvi:                Error 126: The specified module could not be found.
!!!  dvi:                Error 126 loading CoInstaller(wwaninst.dll,WwanDeviceClassCoInstaller)
!!!  dvi:                Error 126: The specified module could not be found.
     dvi:                No CoInstallers found

If you go into a normal Windows installation and into C:\windows\system32, you will notice that the file wlaninst.dll is there so the above error might puzzle you for it says that it could not find the module but if you go to the same location in WindowsPE(X:\windows\system32), you will notice that the file wlaninst.dll is not present. This module is specific to a FULL OS and is not a part of WindowsPE.

The conclusion to derive from this is that the driver that you are installing is for a Full Fledged Operating System and not for Windows PE. To get this to work, you need to download the driver for WindowsPE and refresh deployment share to get the new WIM images with the drivers. Different vendors have different versions of PE driver packs and a few of them can be downloaded from the links below:

(NOTE: The links below are subject to change. There are also several other vendors which are not covered by the list below)

HP: https://ftp.hp.com/pub/caps-softpaq/cmit/HP_WinPE_DriverPack.html

DELL : https://en.community.dell.com/techcenter/enterprise-client/w/wiki/2065.dell-command-deploy-driver-packs-for-enterprise-client-os-deployment

LENOVO: https://support.lenovo.com/in/hi/documents/sccm-index

After you download these driver packs, you need to refresh the deployment share and import the BOOT.WIMs again. If you have the correct driver pack, you should not receive this problem again.

TIP: For Advanced users: If you are an Advanced user of MDT, you probably have your own style of maintaining the deployment share. What I find easier to manage is to keep the deployment share as clean as I can. I prefer creating a different folder within the drivers folder for BootDrivers as below

I then create a Selection Profile by the same name. A selection profile can be created under DeploymentWorkbench>Advanced Configuration>New Selection Profile and select the drivers folder BootDrivers

And then select the BootDrivers Selection profile in the Deployment Share Properties for the x86 and x64 LiteTouchPE

At this point I select the checkbox to completely regenerate the boot images for the simple purpose that I want it to spend less time to remove the old drivers from LiteTouchPE. I then import the new Boot Images to my WDS.

Hope this Helps

Anirudh Gandhi

Support Escalation Engineer |Microsoft Windows Core

Disclaimer : This information is provided ‘as-is’ with no warranties

Hello, My name is Anirudh and today we will be talking about a problem where the Cluster Name Object (CNO) fails to come online in the Failover Cluster Manager.

If you are looking for steps to follow on a general CNO failure, please follow the steps mentioned in the blog https://blogs.technet.microsoft.com/askcore/2012/03/27/why-is-the-cno-in-a-failed-state/ . The steps mentioned below are specific to the situation.

When trying to bring the CNO online, it fails. When we try to repair the CNO, it gives the Event ID as below

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          3/3/2016 9:21:22 PM
Event ID:      1207
Task Category: Network Name Resource
Level:         Error
User:          SYSTEM
Computer:      2012R2Node1.contoso.com
Description: Cluster network name resource 'Cluster Name' cannot be brought online. The computer object associated with the resource could not be updated in domain 'contoso.com' for the following reason:
Unable to update password for computer account.
The text for the associated error code is: The password does not meet the password policy requirements. Check the minimum password length, password complexity and password history requirements.
The cluster identity 'Cluster2012$' may lack permissions required to update the object. Please work with your domain administrator to ensure that the cluster identity can update computer objects in the domain.

When you generate cluster logs for this time for the last 2 minutes using , it gives an error

000011c4.00003922::2016/03/03-21:21:14.760 ERR  [RES] Network Name <Cluster Name>: Unable to update password for computer account Cluster2012 on DC \\DC.contoso.com, status 2245.

000011c4.00003922::2016/03/03-21:21:14.766 WARN  [RES] Network Name <Cluster Name>: Automatic Password rotation failed with status 2245.Will retry in  2245 seconds
000011c4.00003922::2016/03/03-21:21:14.776 INFO  [RES] Network Name <Cluster Name>: TimerQueueTimer rescheduled to fire after 900 secs

Error 2245 translates to

Which is pretty much what we see in the Event logs. This could be due to the User Account Control property set incorrectly in the AD. When you open the properties of the computer object in Active Directory, you see the value of userAccountControl set incorrectly.

If there is a value over there which translates to PASSWD_NOTREQD, it needs to be changed to 4096 (HEX: 0x1000) which translates to WORKSTATION_TRUST_ACCOUNT before attempting the repair of the CNO.

The repair then should reset the password for the CNO and help bring it online.

Hope this helps.

Anirudh Gandhi

Support Escalation Engineer | Microsoft Windows Core

Disclaimer : This information is provided ‘as-is’ with no warranties

Hello everyone! My name is Palash Acharyya and today I am going to talk about a support case I worked where it was determined that a vendor’s DSM was a preferred solution instead of MSDSM when their storage was used in an environment consisting of virtual Fibre Channel Adapters based VMs running on Windows Server 2012/2012 R2.

Summary:

When Live Migrating a VM from one host to another, Hyper-V does not maintain the VM’s MPIO path’s TPGID/RTPID. Unknown to the VM’s DSM (e.g. MSDSM), a given path may now be in a different target port group. This can cause the VM to incorrectly think a Standby path is actually Active Optimized (and vice-versa). I/O can then get routed down a Standby path. This can lead to significantly delayed I/O performance within the VM. In a worst case scenario, Live Migration failures may also occur.

We take an example of a certain Storage vendor over here in this case. The issue is resolved in their updated DSM which monitors for TPGID/RTPID changes in the VM. Customer can contact the DSM vendor for specifics.

At times, we might see significantly decreased I/O performance in the VM after a Live Migration. On a rare occasion, we might also see live migration failures. We frequently see TPGID/RTPID mismatch which is the root cause of this issue (detailed below). Typically, there is no error message other than the I/O performance to the SAN is degraded. After a live migration, mpclaim reports the following for MPIO Disk 0:

C:\Users\Administrator>mpclaim -s -d 0

MPIO Disk0: 08 Paths, Round Robin with Subset, Implicit Only
Controlling DSM: Microsoft DSM
SN: ED8C7618566F92206C9CE900A9C2D800
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID             State             SCSI Address     Weight
------------------------------------------------------------
0000000077060001    Standby           006|000|001|000   0
        TPG_State : Standby          ,TPG_Id: 1, : 4

0000000077060000    Active/Optimized  006|000|000|000   0
        TPG_State : Active/Optimized ,TPG_Id: 2, : 8

0000000077050001    Standby           005|000|001|000   0
        TPG_State : Standby          ,TPG_Id: 1, : 2

0000000077050000    Active/Optimized  005|000|000|000   0
        TPG_State : Active/Optimized ,TPG_Id: 2, : 6

0000000077040001    Active/Optimized  004|000|001|000   0
        TPG_State : Active/Optimized ,TPG_Id: 2, : 5

0000000077040000    Standby           004|000|000|000   0
        TPG_State : Standby          ,TPG_Id: 1, : 1

0000000077030001    Active/Optimized  003|000|001|000   0
        TPG_State : Active/Optimized ,TPG_Id: 2, : 7

0000000077030000    Standby           003|000|000|000   0
        TPG_State : Standby          ,TPG_Id: 1, : 3

Above you see eight paths to the volume. Four Active Optimized and four Standby paths. Note the TPG_Id values. These values never change during the lifetime of the volume (at least not with MSDSM). After a Hyper-V Live Migration, we noticed that some of those TPG_Id values were no longer correct. So, the vendor's developers wrote an in-house tool to submit an inquiry VPD 83h request down each path and display the “true” TPG_Id. When tested with MSDSM, we find the following:

----------------------------------------------
System Disk, Path ID,  SCSI Address,    TPG_Id
----------------------------------------------
          2, 77060001, 006|000|001|000, 1:4
          2, 77060000, 006|000|000|000, 2:8
          2, 77050001, 005|000|001|000, 1:2
          2, 77050000, 005|000|000|000, 2:6
          2, 77040001, 004|000|001|000, 2:5
          2, 77040000, 004|000|000|000, 1:1
          2, 77030001, 003|000|001|000, 1:3
          2, 77030000, 003|000|000|000, 2:7

Above we see that Path ID 77030001 and 77030000 are now out of sync. The DSM thinks Path ID 77030001 is at TPG_Id 2:7 but it’s now at 1:3. Likewise, the DSM thinks Path ID 77030000 is at TPG_Id 1:3 but it’s now at 2:7. This will cause the DSM to incorrectly route I/O down a Standby path about 25% of the time (i.e. 1 of 4 active paths are out of sync).
Both Hyper-V hosts running Server 2012 R2 Standard with all updates. Guest VM failures seen with 2008 R2, 2012, or 2012 R2.

Terms used above:

DSM: DSM or Device Specific Module incorporates knowledge of the manufacturer's hardware. It interacts with the MPIO driver.

TPGID/RTPID: Target Port Group ID

More Information:

Inside Page 18 of publicly available "MPIO Users Guide for Windows Server 2012", it is very clearly mentioned that:

Determining whether to use the Microsoft DSM vs. a Vendor’s DSM

To determine which DSM to use with your storage, refer to information from your hardware storage array manufacturer. Multipath solutions are supported as long as a DSM is implemented in line with logo requirements for MPIO. Most multipath solutions for Windows today use the MPIO architecture and a DSM provided by the storage array manufacturer. You can use the Microsoft DSM provided by Microsoft in Windows Server 2012 if it is also supported by the storage array manufacturer. Refer to your storage array manufacturer for information about which DSM to use with a given storage array, as well as the optimal configuration of it.

NOTE: Multipath software suites available from storage array manufacturers may provide an additional value-add beyond the implementation of the Microsoft DSM because the software typically provides auto-configuration, heuristics for specific storage arrays, statistical analysis, and integrated management. We recommend using the DSM provided by the hardware storage array manufacturer to achieve optimal performance because the storage array manufacturer can make more advanced path decisions in their DSM that are specific to their array, which may result in quicker path failover times.

The default output of Get-MPIOSetting will have PathVerifyEnabled set to 0. While we can change the value in the registry, some vendors do it as a part of the installation.

More about MPIO timers: https://technet.microsoft.com/en-us/library/ee619749(v=ws.10).aspx

MPIO User’s Guide for Windows Server 2012 : https://www.microsoft.com/en-us/download/details.aspx?id=30450

Palash Acharyya

Support Escalation Engineer | Microsoft Windows Core

Disclaimer : This information is provided ‘as-is’ with no warranties

Hi Everyone, Today I am going to talk about one of the most common issues that we see when we talk about patching compliance of cluster nodes. Sometimes, when you run a validation, you see that the nodes complain about missing software updates even though when you scan your machine against your patching tool, it shows ‘No Software Updates available’. Windows Updates shows similar results.

The validation report may look like

In this case, when you check the files updated by KB056572, KB952004 and KB951066, you would see that the file is already of a later version. Also, when you try to install any of these updates, it says:

While this is true, it doesn’t help you justify the warning in the cluster validation report.

So why does this happen?

This happens because the cluster gets the list of install updates by issuing a WMI query which is similar to what gets issues when you run ‘Get-Hotfix’ in PowerShell and does a compare of both to get this information. Even though you might have a different version of the update now, at some point you would have had different update levels on the nodes. Please note that this was a deviation from Microsoft’s recommendation for failover clustering that the nodes have the same software updates. Also, this is the reason that it is a warning in cluster validation and is not registered as an error.

How to validate if you are hitting a false positive?

1. Check the KB number in the validation. Let’s assume it’s KBXXXXXX.
2. Go to https://support.microsoft.com/kb/XXXXXX where XXXXXX is the KB number.
3. Check the column that says ‘File Version’ and check the tab Version over there.
4. Go to C:\windows\system32\drivers or C:\windows\system32 and go to Properties>Details.
5. If the version on the File>= Version in the KB, it is a false positive and can be ignored.

Note: Starting Windows Server 2012, Cluster checks for the file version as well. You would not receive this issue on Windows Server 2012. This was because not everyone followed the recommendation of ‘Same software update level’ at all times.

Hope this helps.

Anirudh Gandhi

Support Escalation Engineer | Microsoft Windows Core

Disclaimer : This information is provided ‘as-is’ with no warranties