Raghavendran KR

Unable to install Roll Update on Exchange servers. Update Rollup Setup reverts to "Rolling Back Actions"

Raghavendran KR — Tue, 20 Feb 2018 12:08:41 GMT

Recently came up with few cases where the customer were unable to install the Rollup Update on Exchange servers. The Update Rollup Setup would revert to "Rolling Back Actions" screen after the "Stopping Services" screen. We started the update from Admin CMD using the following cmdlet C:\Sources\Exchange 2010 SP3 UR 18>msiexec /update c:\temp\Exchange2010-KB4018588-x64-en.msp /lxv* c:\temp\ru18.log We could see the following error in the Exchange Setup log CAQuietExec: Error 0x80070001: Command line returned an error. CAQuietExec: Error 0x80070001: CAQuietExec Failed CustomAction CA_ROLLBACK_SAVEDATA_STOP_SERVICES returned actual error code 1603 but will be translated to success due to continue markingMSI (s) (E0:24) [05:57:33:800]: Executing op: ActionStart(Name=CA_CUSTOMER_PREPATCH_INSTALL,,) MSI (s) (E0:24) [05:57:33:800]: Executing op: ActionStart(Name=CA_CUSTOMER_PATCH_ROLLBACK,,) MSI (s) (E0:24) [05:57:33:800]: Executing op: CustomActionRollback(Action=CA_CUSTOMER_PATCH_ROLLBACK,ActionType=1345,Source=BinaryData,Target=CAQuietExec,CustomActionData="C:\Program Files\Microsoft\Exchange Server\V14\\bin\QuietExe.exe" "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" " -Version 2.0 -command . 'C:\Program Files\Microsoft\Exchange Server\V14\\scripts\customization\CustomPatchInstallerActions.ps1' PatchRollbackActions") MSI (s) (E0:80) [05:57:33:847]: Invoking remote custom action. DLL: C:\Windows\Installer\MSI4D10.tmp, Entrypoint: CAQuietExec CAQuietExec: Error 0x80070001: Command line returned an error. CAQuietExec: Error 0x80070001: CAQuietExec Failed MSI (s) (E0:24) [05:57:34:923]: Executing op: End(Checksum=0,ProgressTotalHDWord=0,ProgressTotalLDWord=0) MSI (s) (E0:24) [05:57:34:923]: Error in rollback skipped. Return: 5 CustomAction CA_CUSTOMER_PATCH_ROLLBACK returned actual error code 1603 but will be translated to success due to continue marking MSI (s) (E0:24) [05:57:34:939]: Note: 1: 2318 2: MSI (s) (E0:24) [05:57:34:939]: No System Restore sequence number for this installation. MSI (s) (E0:24) [05:57:34:939]: Unlocking Server MSI (s) (E0:24) [05:57:35:001]: PROPERTY CHANGE: Deleting UpdateStarted property. Its current value is '1'. MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Note: 1: 2262 2: InstallExecuteSequence 3: -2147287038 MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Note: 1: 2262 2: InstallExecuteSequence 3: -2147287038 MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Note: 1: 2262 2: InstallExecuteSequence 3: -2147287038 MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Transforming table InstallExecuteSequence. MSI (s) (E0:24) [05:57:35:017]: Note: 1: 2262 2: InstallExecuteSequence 3: -2147287038 Action ended 5:57:35: INSTALL. Return value 3. MSI (s) (E0:24) [05:57:35:017]: Product: Microsoft Exchange Server - Update 'Update Rollup 18 for Exchange Server 2010 Service Pack 3 (KB4018588) 14.3.361.1' could not be installed. Error code 1603. Additional information is available in the log file c:\temp\exchange2010KB.log. MSI (s) (E0:24) [05:57:35:017]: Windows Installer installed an update. Product Name: Microsoft Exchange Server. Product Version: 14.3.123.4. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: Update Rollup 18 for Exchange Server 2010 Service Pack 3 (KB4018588) 14.3.361.1. Installation success or error status: 1603. We could see that Exchange setup was trying to use PowerShell Version 2.0 but due to some issues it could not. We checked the version of PowerShell PS C:\Windows\system32> $psversiontable Name Value ---- ----- PSVersion 4.0 WSManStackVersion 3.0 SerializationVersion 1.1.0.1 CLRVersion 4.0.30319.18444 BuildVersion 6.3.9600.18728 PSCompatibleVersions {1.0, 2.0, 3.0, 4.0} PSRemotingProtocolVersion 2.2 We could see PowerShell Version 4.0 was used due to which Exchange setup was reverting to “Rolling Back Actions” since Exchange is not compatible with Powershell 3.0 and above. Process Monitor also showed that it was looking at the file “powershell.exe.activation_config” in following location "C:\Program Files (x86)\Common Files\NetApp\powershell.exe.activation_config" We moved the following file, and restart the SnapDrive Services on the box. Based on the Symptoms, NetAPP's SnapDrive custom config file was forcing PowerShell to run as 4.0 vs 2.0. Exchange 2010 setup runs PowerShell scripts that are incompatible with PowerShell 3.0 and above, since NETAPP was forcing 3.0 via it's custom PowerShell config file, exchange setup scripts fail. We managed to install the RU on the box. -Raghav

Jetstress testing failed. Error: Faulty performance counter paths: \Network Interface()\. Ensure that all the performance counters are working correctly with System Monitor. Error: The specified instance is not present.

Raghavendran KR — Wed, 17 Jan 2018 09:57:46 GMT

I came up with a case last week where the Jetstress test was failing with below error: Jetstress testing failed. Error: Faulty performance counter paths: \Network Interface(*)\*. Ensure that all the performance counters are working correctly with System Monitor. Error: The specified "object was not found on the computer." What is Jetstress? Jetstress is used to verify the performance and stability of a disk subsystem prior to putting an Exchange 2013 or Exchange 2016 server into production. Jetstress helps verify disk performance by simulating Exchange disk Input/Output (I/O) load. Specifically, Jetstress simulates the Exchange database and log file loads produced by a specific number of users. You use Performance Monitor, Event Viewer, and ESEUTIL in conjunction with Jetstress to verify that your disk subsystem meets or exceeds the performance criteria you establish. After a successful completion of the Jetstress Disk Performance and Stress Tests in a non-production environment, you will have ensured that your Exchange disk subsystem is adequately sized (in terms of performance criteria you establish) for the user count and user profiles you have established. It is highly recommended that the Jetstress user read through the tool documentation before using the tool. You can download the Exchange 2013\2016 Jetstress tool from Here Based on the error description, we could see there was some issue with Network Interface counter in Perfmon. We launched the Perfmon from the server and got the below error "Unable to add these counters: \Network Interface(*)\Bytes Total/sec" We ran the following command to query the performance counter service information lodctr /q We could see the TCPIP performance counter was disabled. [Tcpip] Performance Counters (Disabled) DLL Name: %SystemRoot%\System32\Perfctrs.dll Open Procedure: OpenTcpIpPerformanceData Collect Procedure: CollectTcpIpPerformanceData Close Procedure: CloseTcpIpPerformanceData We used the following command to enable the TCPIP performance counter lodctr /e:tcpip We were now able to run the JetStress without any issue. Happy Troubleshooting!!!! -Raghav

Unable to install Cumulative Update on the Exchange server 2013 server. Error: Couldn't remove product with code c3f10d8c-bd70-4516-b2b4-bf6901980741

Raghavendran KR — Wed, 17 Jan 2018 09:34:38 GMT

I recently came up with a case where the customer was not able to install CU18 on a Exchange 2013 server. We were getting the following error while installing the CU Error: ======= Mailbox role: Transport service FAILED The following error was generated when "$error.Clear(); uninstall-MsiPackage -ProductCode '{C3F10D8C-BD70-4516-B2B4-BF6901980741}' -LogFile ($RoleSetupLoggingPath + "\UninstallASDat.msilog")" was run: "Microsoft.Exchange.Configuration.Tasks.TaskException: Couldn't remove product with code c3f10d8c-bd70-4516-b2b4-bf6901980741. The installation source for this product is not available. Verify that the source exists and that you can access it. Error code is 1612. ---> System.ComponentModel.Win32Excep tion: The installation source for this product is not available. Verify that the source exists and that you can access it --- End of inner exception stack trace --- at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl) at Microsoft.Exchange.Management.Deployment.UninstallMsi.RemoveProductCode(Guid productCode) at Microsoft.Exchange.Management.Deployment.UninstallMsi.InternalProcessRecord() at Microsoft.Exchange.Configuration.Tasks.Task.b__b() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)". We could see the following error in the Exchange setup logs: ============================================ Type library exporter warning processing 'Microsoft.Exchange.Data.Transport.Interop.IProxyCallback.SetWriteStream(writeStream), Microsoft.Exchange.Transport.Agent.ContentFilter.ComInterop'. Warning: Type library exporter could not find the type library for 'System.Runtime.InteropServices.ComTypes.IStream'. IUnknown was substituted for the interface. Assembly exported to 'C:\Program Files\Microsoft\Exchange Server\V15\Bin\ComInterop.tlb', and the type library was registered successfully [01/06/2018 01:31:27.0669] [2] Process standard error: [01/06/2018 01:31:27.0681] [2] Ending processing register-ComInteropTLB [01/06/2018 01:31:27.0689] [1] Executing: uninstall-MsiPackage -ProductCode '{C3F10D8C-BD70-4516-B2B4-BF6901980741}' -LogFile ($RoleSetupLoggingPath + "\UninstallASDat.msilog") [01/06/2018 01:31:27.0705] [2] Active Directory session settings for 'Uninstall-MsiPackage' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC.domain.com', Preferred Global Catalog: 'DC.domain.com', Preferred Domain Controllers: '{ 'DC.domain.com' }' [01/06/2018 01:31:27.0706] [2] User specified parameters: -ProductCode:'c3f10d8c-bd70-4516-b2b4-bf6901980741' -LogFile:'C:\ExchangeSetupLogs\UninstallASDat.msilog' [01/06/2018 01:31:27.0706] [2] Beginning processing uninstall-MsiPackage [01/06/2018 01:31:27.0723] [2] Removing MSI package with code 'c3f10d8c-bd70-4516-b2b4-bf6901980741'. The CU was looking for Exchange 2007 Anti-Spam agents file. ASDat.MSI ASEntIRS.MSI In order to fix the issue, we used the Msizap.exe file from the Windows SDK kit. We downloaded the Windows SDK kit from following location https://www.microsoft.com/en-us/download/details.aspx?id=6510 We then pointed the command prompt to the location of Msizap.exe and executed the following command to all the components of Exchange 2007 Anti-Spam from the box. Msizap.exe T {C3F10D8C-BD70-4516-B2B4-BF6901980741} Msizap.exe T {14F288C7-C695-40D5-971D-8890605C6040} Msizap.exe T {93FCFF43-49E2-4AE5-9AD4-0256878AB886} We once again started the CU update and it was successfully installed. Hope this helps !!! -Raghav

Unable to access attachment preview post integration of Exchange 2016 and Office Online Server

Raghavendran KR — Sat, 06 May 2017 21:48:12 GMT

Scenario: We were getting following error while a user from Exchange 2016 was trying to see the attachment preview after Exchange 2016 and OOS integration. An error occurred while the document preview was being created. Please try again later While checking the application logs we saw the following error in Event viewer: Log Name: Application Source: MSExchange OAuth Date: 4/13/2017 4:04:01 PM Event ID: 2004 Task Category: Configuration Level: Warning Keywords: Classic User: N/A Computer: EXC2016.contoso.com Description: Unable to find the certificate with thumbprint DB50F584B3D9175702573349D29079A3D01CE712 in the current computer or the certificate is missing private key. The certificate is needed to sign the outgoing token. Log Name: Application Source: MSExchange Certificate Deployment Date: 4/13/2017 3:59:01 PM Event ID: 2005 Task Category: General Level: Warning Keywords: Classic User: N/A Computer: EXC2016.contoso.com Description: Federation or Auth certificate not found: DB50F584B3D9175702573349D29079A3D01CE712. Unable to find the certificate in the local or neighboring sites. Confirm that the certificate is available in your topology and if necessary, reset the certificate on the Federation Trust to a valid certificate using Set-FederationTrust or Set-AuthConfig. The certificate may take time to propagate to the local or neighboring sites. Run the following command to check if the OAuth Certificate is present or not in the exchange server. (Get-AuthConfig).currentcertificateThumbprint | Get-ExchangeCertificate A special Rpc error occurs on server EXC2016: The certificate with thumbprint DB50F584B3D9175702573349D29079A3D01CE712 was not found. + CategoryInfo : NotSpecified: (:) [Get-ExchangeCertificate], InvalidOperationException + FullyQualifiedErrorId : [Server= EXC2016,RequestId=0db72699-4351-447b-8872-6f2cfc38db38,TimeStamp=9/12/201 6 3:09:29 PM] [FailureCategory=Cmdlet-InvalidOperationException] CC821A5F,Microsoft.Exchange.Management.SystemConf igurationTasks.GetExchangeCertificate + PSComputerName : EXC2016.contoso.com FIX: We ran the following command on the Exchange server New-ExchangeCertificate -server "EXC2016" -PrivateKeyExportable $true Set-AuthConfig -CertificateThumbprint "New Thumbprint" -Server "EXC2016" We ran IISRESET on the Exchange server. We tried and were able to access attachment preview. Note: OAuth certificate is generated at the time of first Exchange server installation in org and same certificate is copied to all subsequent servers. You have to create the certificate by specifying only one Exchange server, then the certificate is copied to all the Exchange server using Microsoft Exchange Service Host service. -Raghavendran KR

Hello world!

Varsha G — Wed, 28 Sep 2016 05:33:13 GMT