CU10 update problem

Marcel Schneider 11 Reputation points
2021-07-08T16:28:15.237+00:00

Hi there, 2 Exchange 2019 standard servers on server OS 2019, hostnames snmta20 and snmta21, update from CU9 to CU10 worked on snmta20. not on snmta21 despite may trials. Both are dual stack, IPv6 tested and working. Below a short summary of the findings:
* Always error at end of step 12/6 Mailbox role: Mailbox service
* SNMTA21 IS IN MAINTENANCE MODE, snmta20 is in normal mode
• Error message from CU10 update process: "System.InvalidOperationException: Failed to mount database "MBX-DB-02". Error: An Active Manager operation failed. Error: Operation failed with message: Error 0x5 (Access is denied) from cli_MountDatabase3 [Server: SNMTA20.SNNET.CH]
• Database redundancy: SNMTA21: The RPC to retrieve the status of active database copy 'MBX-DB-01\SNMTA20' failed. Error: A server-side administrative operation has failed. Operation failed with message: Error 0x5 (Access is denied) from RpccGetCopyStatusEx4 [Server: SNMTA20.SNNET.CH]
• All 3 MBX-DB’s on SNMTA20 as seen from SNMTA21 have status SERVICEDOWN and thus cannot be accessed. From SNMTA20 all 3 MBX-DB’s look good, 3 mounted (SNMTA20), 3 healthy (SNMTA21)
• Event 2280 IIS-W3SVC-WP: The Module DLL C:\Exchange2019\V15\FrontEnd\HttpProxy\bin\exppw.dll failed to load
• Event 5139 WAS: A listener channel for protocol 'http' in worker process '23044' serving application pool 'DefaultAppPool' reported a listener channel failure.
• Service “Microsoft Exchange Search Host Controller” is running
• All server components of SNMTA20 are active, all on SNMTA21 are inactive
• All server components of SNMTA21 cannot be set into Active state, possibly due to maintenance mode.

Assumed a permission problem and have therefore added Organization Management with Full Control to all three MBX-DB directories and MBX EDB files. Also gave Full Control to Authenticated Users. Did not help. Thank you for any suggestions, Marcel

Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
6,333 questions

7 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Marcel Schneider 11 Reputation points
    2021-08-03T06:58:52.403+00:00

    Hi Kael Yao,

    How are things going now?

    Despite your bad advice: All 3 MTA's are running. The culprit was not no. 21, an entry in a domain controller was missing. Thus, the complete rebuild of no. 21 was a major waste of time. Because you appear as an MS-employee, I trusted you. That was a big mistake and it will never ever happen again. And, btw., the question regarding rehydration you did not even bother to answer.

    Regards, Marcel

    1 person found this answer helpful.
  2. Kael Yao-MSFT 24,596 Reputation points Microsoft Vendor
    2021-07-09T07:32:07.18+00:00

    Hi Marcel.

    Thanks for the detailed information.

    Here are a few more questions:

    1. Are you using the same account on both servers?
    2. Will it return permission errors when running other EMS commands on SNMTA21?

    If other EMS commands can be run successfully,
    since all databases are currently active on SNMTA20, if possible, please remove the passive copies on SNMTA21 and remove SNMTA21 from DAG.
    And retry the upgrade and see if you can upgrade successfully.

    If the response is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

  3. Marcel Schneider 11 Reputation points
    2021-07-12T06:07:03.397+00:00

    Hi Kael Yao,

    Why have my last answers disappeared. E.g. the last you have answered? You must have seen it. Also cannot see your answer but try to comment it from e-mail:

    Did you successfully remove DAG?

    Yes. The other machine is now the only MTA, all necessary DNS entries have been changed. But - as mentioned - it writes now every 15 minutes event 4999 MSExchangecommon i

    Watson report about to be sent for process id: 8820, with parameters: E12, c-RTL-AMD64, 15.02.0922.005, MSExchangeHMWorker, M.Exchange.Diagnostics,
    .0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, PerformActualLog, M.E.Diagnostics.BlockedDeserializeTypeException, 1193763487, 15.02.0922.005.
    ErrorReportingEnabled: True
    exData=|exHResult=|exStacktrace=at Microsoft.Exchange.Diagnostics.DeserializationTypeLogger.PerformActualLog(Dictionary`2 hits)
    at Microsoft.Exchange.Diagnostics.DeserializationTypeLogger.ProcessCachedHits()
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.TimerQueueTimer.CallCallback()
    at System.Threading.TimerQueueTimer.Fire()
    at System.Threading.TimerQueue.FireNextTimers()|exTargetSite=Microsoft.Exchange.Diagnostics, Version=15.0.0.0, Culture=neutral,
    PublicKeyToken=31bf3856ad364e35 PerformActualLog|exSource=|exMessage=|exComplete=

    please reinstall the operating system and install CU10 directly

    Why uninstall OS?

    Regards. Marcel

  4. Marcel Schneider 11 Reputation points
    2021-07-09T11:30:29.063+00:00

    Hi Kael Yao,

    MTA21 is now again in dual stack. Error when connecting Exchange Management Shell to SNMTA21:
    VERBOSE: Connecting to SNMTA21.xxxx.
    New-PSSession : [snmta21.xxxx] Connecting to remote server snmta21.xxxx failed with the following error
    message : The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from
    the destination computer. The content type is absent or invalid. For more information, see the
    about_Remote_Troubleshooting Help topic.
    At line:1 char:1

    • New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Micr ...
    • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    • CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
      gTransportException
    • FullyQualifiedErrorId : -2144108297,PSSessionOpenFailed

    And when trying to open EMC these errors occur:

    Server Error in '/owa' Application.

    Configuration Error
    Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

    Parser Error Message: Could not load file or assembly 'Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

    Source Error:

    Line 67: the compiler. All assemblies in the GAC and owa\bin are referenced automatically.
    Line 68: -->
    Line 69: <add assembly="Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, publicKeyToken=31bf3856ad364e35" />
    Line 70: <add assembly="Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    Line 71: <add assembly="Microsoft.Exchange.Clients.Common, Version=15.0.0.0,Culture=neutral, publicKeyToken=31bf3856ad364e35" />

    Source File: C:\Exchange2019\V15\FrontEnd\HttpProxy\owa\web.config Line: 69

    Assembly Load Trace: The following information can be helpful to determine why the assembly 'Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' could not be loaded.

    WRN: Assembly binding logging is turned OFF.
    To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
    Note: There is some performance penalty associated with assembly bind failure logging.
    To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

    Best, Marcel

  5. Marcel Schneider 11 Reputation points
    2021-07-19T07:31:00.823+00:00

    Hi Kael Yao,
    Nothing has been resolved. The current situation: 3 MTA's (with get-clientaccessservice) 19, 20, 21. 21 is the one used as MX 10 and where autodiscovers are pointing to. 21 exhibited the "Failed to mount database "MBX-DB-02". Error: An Active Manager operation failed. Error: Operation failed with message: Error 0x5 (Access is denied) from cli_MountDatabase3 [Server: SNMTA20.SNNET.CH]" errors. It worked for 4.5 years, only with CU10 these error occurred and I plan to implement your most drastic solution: To completely rebuild it. No. 20 has taken over the role of 21 and 19 is a new Exchange 2019 license, built on a previously spare OS 2019 machine.

    No. 19 never worked as expected. Firing up EAC shows error 500, it looks for a non-present resource. I login with Admin to 19 and assume it looks for an admin mailbox on its newly created database. But cannot add admin to this MBX because there is already one in the 3 legacy MBX'es. Have tried to create a new DAG with 19 and 20. No luck, 19 never became active. And since these efforts, get tons of 3002 events, failed MsExchange BackEndRehydration. They complain about protocols mapi/emsmdb, mapi/nspi, Autodiscover and Microsoft-Server-ActiveSync. Never seen such events. My assumption is that probably NT Authority/System service account rights are missing for ms-Exch-EPI-Token-Serialization. Do not understand what it wants to "hydrate".

    The plan is now to completely decommission 21 and rebuild it. Then enclose it in a DAC with 20 and change DNS RR's to 21 as before. After that probably have to decommission and build 19 again. Exchange can be a mess and it is now one here.
    Regards, Marcel