Exchange 2019 CU10 database switch / failover is not working between two nodes in single dag

Mashhour 26 Reputation points
2021-09-13T06:22:20.22+00:00

Hello all;
I have Exchange 2019 organization which consists of two nodes exchange 2019 in a DAG.
the two exchange servers are in one AD site, AD itself has one single AD site.
after upgrading the two Exchange servers to CU10. I face an issue in the failover / switch over the DBs between the two nodes, actually the DBs AP is distributed between the two Ex2019 servers. I uninstalled all 3rd party apps including the file level antivirus. and restarted both servers multiple times. I have one network adapter on each server, no replication dedicated adapter, the servers are on the same subnet, and are hosted on VMWare, I checked both scenarios where the servers are on the same host or on different VMWare host. all got the same failure results. now all DBs are active in the one of the servers, running the script ".\RedistributeActiveDatabases.ps1 -DagName DAG -BalanceDbsByActivationPreference". the script completes successfully, the DBs are balanced for a second then the DBs failover to the second server, multiple errors are there including the following:
MSExchangeIS error 1001 in application log in the server that is not accepting mounting DBs
"Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (ProcessId perf counter (0) does not match actual process id (20172).) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.Common.Globals.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.PerformanceCounterFactory.CreateDatabaseInstance(StoreDatabase database)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Globals.DatabaseMounting(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StartupShutdown.Globals.DatabaseMountingHandler(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.FinishMount(Context context, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.ActivatePassive(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics, ClientType clientType)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass37_0.<EcExecute>b__0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatchT
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute()
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)
at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)
at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* )
at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* )
at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, _GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut)
)."
this error is followed by MSExchangeIS error 1002 in the app log
"Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: ProcessId perf counter (0) does not match actual process id (20172).
at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters)
at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.Common.Globals.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.PerformanceCounterFactory.CreateDatabaseInstance(StoreDatabase database)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Globals.DatabaseMounting(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StartupShutdown.Globals.DatabaseMountingHandler(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.FinishMount(Context context, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.ActivatePassive(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics, ClientType clientType)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass37_0.<EcExecute>b__0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatchT)."

then followed by MSExchange Common error 4999
"Watson report about to be sent for process id: 20172, with parameters: E12, c-RTL-AMD64, 15.02.0922.013, M.E.Store.Worker, M.E.S.Storage.StoreCommonServices, M.E.S.S.S.PerformanceCounterFactory.CreateDatabaseInstance, M.E.Diagnostics.ExAssertException, 6c3-dumptidset, 15.02.0922.013.
ErrorReportingEnabled: True

the issue is not yet been solved.

Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
6,530 questions
{count} votes

Accepted answer
  1. Zhengqi Lou-MSFT 8,821 Reputation points Microsoft Employee
    2021-09-14T05:54:58.317+00:00

    Hi @Mashhour ,

    Have you tried removing and recreating the DAG?

    the two exchange servers are in one AD site, AD itself has one single AD site.

    Do you mean the DC and the Exchange servers are not in one AD site?

    Out situation is that now the databases are all mounted on one server, assuming it is Server1. When trying to manually mount the database copy on Server2, it failed, is that right?
    I think you could remove the DAG and uninstall the Failover Cluster Service in Server Manager, and then retry creating the DAG or first re-install the Failover Cluster Service and create the DAG.

    As for the Error event 4999, I did a lot research but still have no methods, I will do some tests and if I have any updates, I will post here.

    Best regards,
    Lou


    If the response is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


1 additional answer

Sort by: Most helpful
  1. Sanket Salvi 1 Reputation point
    2021-09-18T02:29:28.243+00:00

    hi @Mashhour

    If the issue is still not fixed. please check the link below.

    https://ingogegenwarth.wordpress.com/2017/02/07/exchange-2016-and-filemappingsize/

    Regards,
    Sanket

    0 comments No comments