Failed install of SU KB5007409 - everything broken

MNM Show 11 Reputation points
2021-12-04T22:00:55.897+00:00

Server 2012 R2 - Exchange Server 2013 CU 23 with the KB5004778

I was doing windows updates through the control panel windows update. It found 3 for windows plus this one for exchange. I selected all of them, installed, and when it indicated it was finished I rebooted. I always set my exchange services to disabled when rebooting, then set them back to auto and start them manually. It just makes server reboots a lot faster.

After reboot I went into windows update control panel and it appeared to have only installed November 9, 2021—KB5007247 (Monthly Rollup) as the other 3 were listed again (or still). At this point my exchange services were all off still. I went ahead and did the other two windows updates and rebooted. Now when I try and start Exchange services most of them fail. They start and immediately stop, no real errors. I haven't tried them all but since I always do them in order I am sure some fail because of previous dependency failures. I can get the first two to start (AD-Top and Anti-Spam) but starting with DAG management many or most fail after that.

Windows Update does not show a failed install of KB5007409. But ServiceControl.log shows this from the timeframe of the original attempt to install all 4 updates through Windows Update:

[08:58:28] * ServiceControl.ps1: 12/4/2021 8:58:28 AM
[08:58:28] Performing service control with options:
[08:58:31] Saving service and registry data
[08:58:31] Saving service state to 'C:\ExchangeSetupLogs\ServiceState.xml'...
[08:58:37] Saving services startup mode.
[08:58:38] Adding to installed roles list: AdminTools
[08:58:38] Adding to installed roles list: ClientAccessMailboxRole
[08:58:38] Adding to installed roles list: Mailbox
[08:58:38] Adding to installed roles list: Bridgehead
[08:58:38] Adding to installed roles list: Mailbox
[08:58:38] Adding to installed roles list: UnifiedMessaging
[08:58:38] Stopping services for the following roles: AdminTools ClientAccess FrontendTransport Bridgehead Mailbox UnifiedMessaging
[08:58:38] Stopping services for 'AdminTools ClientAccess FrontendTransport Bridgehead Mailbox UnifiedMessaging'...
[08:58:39] Stopping service 'WinMgmt'.
[08:58:50] Stopping service 'W3Svc'.
[09:00:20] Stopping service 'SearchExchangeTracing'.
[09:00:21] Stopping service 'pla'.
[09:00:21] Stopping service 'MSExchangeUM'.
[09:00:22] Service 'MSExchangeUM' failed to stop due to error:'Cannot stop MSExchangeUM service on computer '.'.'.
[09:00:22] Unable to stop all services for AdminTools ClientAccess FrontendTransport Bridgehead Mailbox UnifiedMessaging.

I have attempted to install the SU directly using elevated command prompt and ServiceControl.log reports this:

[14:13:44] -----------------------------------------------
[14:13:44] * ServiceControl.ps1: 12/4/2021 2:13:44 PM
[14:13:44] Performing service control with options:
[14:13:48] [Error] System.TypeInitializationException: The type initializer for 'Microsoft.Exchange.Management.PowerShell.CmdletConfigurationEntries' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Exchange.Data.Mapi, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
at Microsoft.Exchange.Management.PowerShell.CmdletConfigurationEntries..cctor()
--- End of inner exception stack trace ---
at Microsoft.Exchange.Management.PowerShell.CmdletConfigurationEntries.get_SetupCmdletConfigurationEntries()
at Microsoft.Exchange.Management.PowerShell.SetupPSSnapIn.get_Cmdlets()
at System.Management.Automation.Runspaces.InitialSessionState.MergeCustomPSSnapIn(PSSnapInInfo psSnapInInfo, CustomPSSnapIn customPSSnapIn)
at System.Management.Automation.Runspaces.InitialSessionState.LoadCustomPSSnapIn(PSSnapInInfo psSnapInInfo)
at System.Management.Automation.Runspaces.InitialSessionState.ImportPSSnapIn(PSSnapInInfo psSnapInInfo, PSSnapInException& warning)
at System.Management.Automation.Runspaces.InitialSessionState.ImportPSSnapIn(String name, PSSnapInException& warning)
at Microsoft.PowerShell.Commands.AddPSSnapinCommand.AddPSSnapIns(Collection`1 snapInList)
at Microsoft.PowerShell.Commands.AddPSSnapinCommand.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()

It seems the original attempt to install windows updates has done something but I am unsure how to fix it. Needless to say, without most of the services being able to start there is little I am able to access for Exchange including the EMS, although not EMS powershell is working.

Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
7,376 questions
0 comments No comments
{count} votes

3 answers

Sort by: Most helpful
  1. MNM Show 11 Reputation points
    2021-12-06T14:08:37.773+00:00

    I have done exactly this. Seemed to work ok. Exchange was working after recovery and except for a few little issues (the cert probably being the worst of them) it was functioning. I once again installed KB5007409 and it has again broken things although less severely then before. Currently ECP and OWA both load a login window and then when I log in they go to a code 500 :-( something went wrong page.

    this page: https://learn.microsoft.com/en-us/exchange/troubleshoot/client-connectivity/owa-stops-working-after-update says that means the update was not run with elevated prompt but I guarantee that is not the case. The update took a while, went through many steps and reported it had succeeded.

    As per that page I tried the update UpdateCas.ps1 and UpdateConfigFiles.ps1 solution and it actually improved the OWA situation, OWA had been loading an error page saying too many redirects and no login screen at all. But needless to say the web front end is still not working.

    All exchange services are running (yay!) and EMS works (also yay!) so I just need to figure out how to get the web pages working.

    2 people found this answer helpful.

  2. Dave Patrick 426.2K Reputation points MVP
    2021-12-04T23:04:43.113+00:00

    The event logs may provide something useful. I'd be tempted to set the services all back to their default or correct startup type and reboot it.


  3. MNM Show 11 Reputation points
    2021-12-05T00:39:12.61+00:00

    This is what I have so far with other Exchange services:

    Microsoft Exchange Active Directory Topology
    -starts and runs normally

    Microsoft Exchange Anti-spam Update
    -starts and runs normally

    Microsoft Exchange DAG Management
    -Fails or immediately stops with:

    MSExchange Common Event 4999
    Watson report about to be sent for process id: 5844, with parameters: E12IIS, c-RTL-AMD64, 15.00.1497.023, MSExchangeDagMgmt, MSExchangeDagMgmt, M.E.C.D.DagComponentManager..ctor, System.TypeInitializationException, 8fde, 15.00.1497.023.
    ErrorReportingEnabled: True

    Microsoft Exchange Diagnostics
    -starts and runs – generates a number of errors that may indicate something:

    MSExchangeDiagnostics Event 1033
    The PerfLog job has crashed. Failed to create analyzer Microsoft.Exchange.Monitoring.ActiveMonitoring.EDS.Analyzers.PerfLogActiveMonitoringAnalyzer. Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Office.Datacenter.ActiveMonitoringLocal, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
    at Microsoft.Exchange.Monitoring.ActiveMonitoring.EDS.Analyzers.PerfLogActiveMonitoringAnalyzer..ctor(IJob job)
    --- End of inner exception stack trace ---
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
    at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    at Microsoft.Exchange.Diagnostics.Service.JobManager.CreateAnalyzer(AnalyzerConfigurationElement configuredAnalyzer, WatermarkedJob newJob)


    MSExchange Common Event 4999
    Watson report about to be sent for process id: 3032, with parameters: E12IIS, c-RTL-AMD64, 15.00.1497.023, M.E.Diagnostics.Service, M.E.M.A.Local.Components, M.E.M.A.E.A.PerfLogActiveMonitoringAnalyzer..ctor, S.Reflection.TargetInvocationException, a41e, 15.00.1497.023.

    ErrorReportingEnabled: True

    Service Control Manager Event 7031
    The Microsoft Exchange Diagnostics service terminated unexpectedly. It has done this 3 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
    Microsoft Exchange EdgeSync
    -starts and runs normally

    Microsoft Exchange Frontend Transport
    -Fails with:

    Service Control Manager Event 7000
    The Microsoft Exchange Frontend Transport service failed to start due to the following error:
    The service did not respond to the start or control request in a timely fashion.

    Microsoft Exchange Health Manager
    -Fails with:

    MSExchangeHM Event 1016
    Microsoft Exchange Health Manager RPC server failed to start with the following exception:
    Could not load file or assembly 'Microsoft.Office.Datacenter.ActiveMonitoringLocal, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

    MSExchange Common Event 4999
    Watson report about to be sent for process id: 3480, with parameters: E12IIS, c-RTL-AMD64, 15.00.1497.023, MSExchangeHMWorker, MSExchangeHMWorker, M.E.A.MonitoringWorker.Run, S.IO.FileNotFoundException, 1519, 15.00.1497.023.
    ErrorReportingEnabled: True

    MSExchangeHMHost Event 1015
    Failed to start the worker process: C:\Program Files\Microsoft\Exchange Server\V15\Bin\MSExchangeHMWorker.exe. The service will be stopped.