Hello all;
I have Exchange 2019 organization which consists of two nodes exchange 2019 in a DAG.
the two exchange servers are in one AD site, AD itself has one single AD site.
after upgrading the two Exchange servers to CU10. I face an issue in the failover / switch over the DBs between the two nodes, actually the DBs AP is distributed between the two Ex2019 servers. I uninstalled all 3rd party apps including the file level antivirus. and restarted both servers multiple times. I have one network adapter on each server, no replication dedicated adapter, the servers are on the same subnet, and are hosted on VMWare, I checked both scenarios where the servers are on the same host or on different VMWare host. all got the same failure results. now all DBs are active in the one of the servers, running the script ".\RedistributeActiveDatabases.ps1 -DagName DAG -BalanceDbsByActivationPreference". the script completes successfully, the DBs are balanced for a second then the DBs failover to the second server, multiple errors are there including the following:
MSExchangeIS error 1001 in application log in the server that is not accepting mounting DBs
"Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (ProcessId perf counter (0) does not match actual process id (20172).) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.Common.Globals.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.PerformanceCounterFactory.CreateDatabaseInstance(StoreDatabase database)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Globals.DatabaseMounting(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StartupShutdown.Globals.DatabaseMountingHandler(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.FinishMount(Context context, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.ActivatePassive(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics, ClientType clientType)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass37_0.<EcExecute>b__0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatchT
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute()
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)
at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)
at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* )
at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* )
at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, _GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut)
)."
this error is followed by MSExchangeIS error 1002 in the app log
"Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: ProcessId perf counter (0) does not match actual process id (20172).
at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters)
at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.Common.Globals.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.PerformanceCounterFactory.CreateDatabaseInstance(StoreDatabase database)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Globals.DatabaseMounting(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StartupShutdown.Globals.DatabaseMountingHandler(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.FinishMount(Context context, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.ActivatePassive(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics, ClientType clientType)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass37_0.<EcExecute>b__0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatchT)."
then followed by MSExchange Common error 4999
"Watson report about to be sent for process id: 20172, with parameters: E12, c-RTL-AMD64, 15.02.0922.013, M.E.Store.Worker, M.E.S.Storage.StoreCommonServices, M.E.S.S.S.PerformanceCounterFactory.CreateDatabaseInstance, M.E.Diagnostics.ExAssertException, 6c3-dumptidset, 15.02.0922.013.
ErrorReportingEnabled: True
the issue is not yet been solved.