'MSExchangeMapiMailboxAppPool' crashes sometimes

Wouter 1 Reputation point
2021-09-29T09:39:03.717+00:00

We have an Exchange 2019 server (CU10) running on Windows server 2019. It has about 800 mailboxes and a part of those mailboxes are accessed through Outlook (2016 / 2019) on RDS servers, so they run in online mode.

The issue we have is that sometimes (about 3 times a week) the 'MSExchangeMapiMailboxAppPool' application pool crashes. The application logs show:

A process serving application pool 'MSExchangeMapiMailboxAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '41584'. The data field contains the error number

All MAPI connections are dropped and at that moment any Outlook that runs in online mode freezes for a few minutes (not responding).

We run Sophos on the Exchange server and all exclusions recommended by Microsoft have been added and AMSI has been disabled as well.

Internet Information Services
Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
7,481 questions
{count} votes

5 answers

Sort by: Most helpful
  1. Wouter 1 Reputation point
    2021-09-30T11:16:47.183+00:00

    A few seconds before the application pool error I can see a few errors in the application logs in the eventviewer:


    Source: MSExchange Common
    ID: 4999

    Watson report about to be sent for process id: 26864, with parameters: E12IIS, c-RTL-AMD64, 15.02.0922.013, w3wp#MSExchangeMapiMailboxAppPool, M.E.Data.Storage, M.E.D.S.CoreAttachmentCollection.OnAfterAttachmentSave, System.ArgumentException, 743f-dumptidset, 15.02.0922.013.
    ErrorReportingEnabled: True


    Source: ASP.NET 4.0.30319.0
    ID: 1325

    An unhandled exception occurred and the process was terminated.

    Application ID: DefaultDomain

    Process ID: 26864

    Exception: System.Runtime.Serialization.SerializationException

    Message: Type 'Microsoft.Exchange.Diagnostics.ExWatson+CrashNowException' in Assembly 'Microsoft.Exchange.Diagnostics, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' is not marked as serializable.

    StackTrace: at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
    at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
    at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
    at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
    at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeObject(Object obj, MemoryStream stm)
    at System.AppDomain.Serialize(Object o)
    at System.AppDomain.MarshalObject(Object o)


    Source: .NET Runtime
    ID: 1026

    Application: w3wp.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.ArgumentException
    at System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)
    at System.Collections.Generic.TreeSet1[[System.Collections.Generic.KeyValuePair2[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].AddIfNotPresent(System.Collections.Generic.KeyValuePair2<Int32,System.__Canon>) at System.Collections.Generic.SortedDictionary2[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Add(Int32, System.__Canon)
    at Microsoft.Exchange.Data.Storage.CoreAttachmentCollection.OnAfterAttachmentSave(Int32)
    at Microsoft.Exchange.Data.Storage.AttachmentPropertyBag.SaveChanges(Boolean)
    at Microsoft.Exchange.Data.Storage.CoreAttachment.Save()
    at Microsoft.Exchange.RpcClientAccess.Handler.Attachment.SaveChanges(Microsoft.Exchange.RpcClientAccess.Parser.SaveChangesMode)
    at Microsoft.Exchange.RpcClientAccess.Handler.RopHandler+<>c__DisplayClass120_0.<SaveChangesAttachment>b__0()
    at Microsoft.Exchange.RpcClientAccess.Handler.ExceptionTranslator.TryExecuteCatchAndTranslateExceptions[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]
    at Microsoft.Exchange.RpcClientAccess.Handler.RopHandlerHelper.CallHandler[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]
    at Microsoft.Exchange.RpcClientAccess.Handler.RopHandlerHelper.CallHandler(Microsoft.Exchange.RpcClientAccess.Parser.IRopHandler, System.Func1<Microsoft.Exchange.RpcClientAccess.Parser.RopResult>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory, System.Func4<Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory,Microsoft.Exchange.RpcClientAccess.ErrorCode,System.Exception,Microsoft.Exchange.RpcClientAccess.Parser.RopResult>)
    at Microsoft.Exchange.RpcClientAccess.Handler.RopHandler.GenericRopHandlerMethod(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, System.Func1<Microsoft.Exchange.RpcClientAccess.Parser.RopResult>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory, System.Func4<Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory,Microsoft.Exchange.RpcClientAccess.ErrorCode,System.Exception,Microsoft.Exchange.RpcClientAccess.Parser.RopResult>)
    at Microsoft.Exchange.RpcClientAccess.Handler.RopHandler.StandardRopHandlerMethod(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, System.Func1<Microsoft.Exchange.RpcClientAccess.Parser.RopResult>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory) at Microsoft.Exchange.RpcClientAccess.Parser.RopSaveChangesAttachment.InternalExecute(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, Microsoft.Exchange.RpcClientAccess.Parser.IRopHandler, System.ArraySegment1<Byte>)
    at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(Microsoft.Exchange.RpcClientAccess.Parser.IConnectionInformation, Microsoft.Exchange.RpcClientAccess.Parser.IRopDriver, Microsoft.Exchange.RpcClientAccess.Parser.ServerObjectHandleTable, System.ArraySegment1<Byte>) at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(System.Collections.Generic.List1<System.ArraySegment1<Byte>>, Microsoft.Exchange.RpcClientAccess.Parser.ServerObjectHandleTable, System.ArraySegment1<Byte>, Int32, Int32, Boolean, Int32 ByRef, Microsoft.Exchange.RpcClientAccess.Parser.AuxiliaryData, Boolean, Byte[] ByRef)
    at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteOrBackoff(System.Collections.Generic.IList1<System.ArraySegment1<Byte>>, System.ArraySegment1<Byte>, Int32 ByRef, Microsoft.Exchange.RpcClientAccess.Parser.AuxiliaryData, Boolean, Byte[] ByRef) at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.Execute(System.Collections.Generic.IList1<System.ArraySegment1<Byte>>, System.ArraySegment1<Byte>, Int32 ByRef, Microsoft.Exchange.RpcClientAccess.Parser.AuxiliaryData, Boolean, Byte[] ByRef)
    at Microsoft.Exchange.RpcClientAccess.Server.RpcDispatch+<>c__DisplayClass49_0.<Execute>b__2()
    at Microsoft.Exchange.RpcClientAccess.Server.RpcDispatch.ExecuteWrapper(System.Func1<ExecuteParameters>, System.Func1<Microsoft.Exchange.RpcClientAccess.RpcErrorCode>, System.Action1<System.Exception>) at Microsoft.Exchange.RpcClientAccess.Server.RpcDispatch.Execute(Microsoft.Exchange.Rpc.IProtocolRequestInfo, IntPtr ByRef, System.Collections.Generic.IList1<System.ArraySegment1<Byte>>, System.ArraySegment1<Byte>, Int32 ByRef, System.ArraySegment1<Byte>, System.ArraySegment1<Byte>, Int32 ByRef, Boolean, Byte[] ByRef)
    at Microsoft.Exchange.RpcClientAccess.Server.WatsonOnUnhandledExceptionDispatch+<>c__DisplayClass32_0.<Microsoft.Exchange.RpcClientAccess.Server.IRpcDispatch.Execute>b__0()
    at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(System.Action, System.Func2<System.Object,Boolean>, System.Action1<System.Object>)

    Exception Info: Microsoft.Exchange.Diagnostics.ExWatson+CrashNowException
    at Microsoft.Exchange.Diagnostics.ExWatson.CrashNow()
    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
    at System.Threading.ThreadHelper.ThreadStart()


    Source: Application Error
    ID: 1000

    Faulting application name: w3wp.exe, version: 10.0.17763.1, time stamp: 0xcfdb13d8
    Faulting module name: KERNELBASE.dll, version: 10.0.17763.2183, time stamp: 0x8e097f91
    Exception code: 0xe0434352
    Fault offset: 0x0000000000039329
    Faulting process id: 0x68f0
    Faulting application start time: 0x01d7b51c6fb1ee51
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\Windows\System32\KERNELBASE.dll
    Report Id: 0a62472a-6a07-48b1-9b7c-a19f297900c3
    Faulting package full name:


    We do not need to do anything to get it working again, it recovers itself (Outlook not responding for a few minutes during this time).

    We use PRTG as monitoring and what I can see there at the time of the error:

    MAPI connections drops to 0, has a spike after that and back to normal as it was before:

    136624-image.png

    There is a spike in IIS Current Connections:

    136625-image.png

    Additionally there is a increase of network traffic and CPU load.


  2. Wouter 1 Reputation point
    2021-10-07T17:19:24.933+00:00

    I did run Debug Diag, but while that is running it makes things worse (Outlook keeps hanging a lot) and the application pool seems to crash faster.

    Anyway, I did get some logs, but didn't find any clue about a cause. There are various errors but for some I cannot find a lot of information online and for others Microsoft notes that these can be ignored.

    A summary of Debug Diag:


    • EXCEPTION SUMMARY *
      |-------------------------------------------------------------------------------------------------------|
      | Count | Exception |
      |-------------------------------------------------------------------------------------------------------|
      | 284 | Microsoft.Exchange.Data.Storage.ObjectNotFoundException |
      | 174 | Microsoft.Mapi.MapiExceptionNotFound |
      | 68 | Microsoft.Exchange.Data.Directory.ValueNotPresentException |
      | 58 | Microsoft.Exchange.RpcClientAccess.Server.LoginPermException |
      | 53 | Microsoft.Exchange.Data.Storage.SaveConflictException |
      | 53 | Microsoft.Mapi.MapiExceptionObjectChanged |
      | 49 | Microsoft.Exchange.RpcClientAccess.RopExecutionException |
      | 26 | Microsoft.Mapi.MapiExceptionNoAccess |
      | 26 | Microsoft.Exchange.Data.Storage.AccessDeniedException |
      | 15 | Microsoft.Exchange.Data.Storage.StoragePermanentException |
      | 14 | Microsoft.Mapi.MapiExceptionPartialCompletion |
      | 14 | Microsoft.Exchange.Data.Storage.PartialCompletionException |
      | 10 | Microsoft.Exchange.Data.StoreObjects.DefaultFolders.Exceptions.DefaultFolderNoParentException |
      | 6 | System.ServiceModel.CommunicationException |
      | 4 | Microsoft.Mapi.MapiExceptionCallFailed |
      | 4 | Microsoft.Exchange.MapiHttp.ContextNotFoundException |
      | 3 | 0XE06D7363 |
      | 3 | System.MissingMethodException |
      | 3 | Microsoft.Exchange.RpcClientAccess.BufferOutOfMemoryException |
      | 2 | Microsoft.Exchange.Collections.TimeoutCache.DuplicateKeyException |
      | 2 | System.IO.IOException |
      | 2 | Microsoft.Exchange.Data.Storage.Extensions.ExtensionNotFoundException |
      | 2 | System.Net.Sockets.SocketException |
      | 1 | Microsoft.Mapi.MapiExceptionNotInQueue |
      | 1 | Microsoft.Exchange.Data.ServerLocator.ServerLocatorClientTransientException |
      | 1 | Microsoft.Exchange.HttpProxy.Routing.Providers.ServerLocatorClientException |
      | 1 | System.Reflection.TargetInvocationException |
      |-------------------------------------------------------------------------------------------------------|

    **** Some exception details were omitted due to the following reasons
    The maximum number of stacks (MAX_CLR_EXCEPTION_STACKS_PER_EXCEPTION_TYPE = 10) for this CLR exception type have been collected: 'Microsoft.Exchange.Data.Storage.SaveConflictException'
    The maximum number of total stacks (MAX_CLR_EXCEPTION_STACKS_TOTAL = 100) for all CLR exception types have been collected)
    The maximum number of stacks (MAX_CLR_EXCEPTION_STACKS_PER_EXCEPTION_TYPE = 10) for this CLR exception type have been collected: 'Microsoft.Exchange.Data.Storage.ObjectNotFoundException'
    The maximum number of unique messages (MAX_VARIANTS_PER_EXCEPTION_TYPE = 20) for this CLR exception type have been collected: 'Microsoft.Mapi.MapiExceptionNotFound'
    The maximum number of stacks (MAX_CLR_EXCEPTION_STACKS_PER_EXCEPTION_TYPE = 10) for this CLR exception type have been collected: 'Microsoft.Exchange.RpcClientAccess.Server.LoginPermException'
    The maximum number of unique messages (MAX_VARIANTS_PER_EXCEPTION_TYPE = 20) for this CLR exception type have been collected: 'Microsoft.Exchange.RpcClientAccess.RopExecutionException'
    The maximum number of unique messages (MAX_VARIANTS_PER_EXCEPTION_TYPE = 20) for this CLR exception type have been collected: 'Microsoft.Mapi.MapiExceptionObjectChanged'
    The maximum number of stacks (MAX_CLR_EXCEPTION_STACKS_PER_EXCEPTION_TYPE = 10) for this CLR exception type have been collected: 'Microsoft.Exchange.Data.Storage.PartialCompletionException'
    The maximum number of unique messages (MAX_VARIANTS_PER_EXCEPTION_TYPE = 20) for this CLR exception type have been collected: 'Microsoft.Mapi.MapiExceptionNoAccess'
    The maximum number of stacks (MAX_CLR_EXCEPTION_STACKS_PER_EXCEPTION_TYPE = 10) for this CLR exception type have been collected: 'Microsoft.Exchange.Data.Storage.AccessDeniedException'
    The maximum number of unique messages (MAX_VARIANTS_PER_EXCEPTION_TYPE = 20) for this CLR exception type have been collected: 'Microsoft.Exchange.Data.Directory.ValueNotPresentException'

    Debugging Overhead Cost:
    Total Elapsed Ticks = 1145015 (100%)
    Total Ticks Spent in Debugger Engine = 2140 (0%)
    Total Ticks Spent in Crash Rule Script = 93675 (8%)
    Successfully detached from process. The target process is no longer being debugged.
    This instance of Dbghost will shutdown in a few seconds.

    Note that the load of the Exchange server is not high, summary:

    • It has 12 cores (average load 40~60%)
    • 80GB memory (average load 50%)
    • Storage is a Nimble all flash SAN
    • VMware cluster fully patched

    In the meantime we are running CU11 and without Sophos.

    I did run the Exchange Health Checker (https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/) which showed all was good.

    We have 4 DB's on seperate partitions (DB logs on seperate partition too), DB's are about 1TB in size. Could the DB size be a problem? We did have the same sizes on Exchange 2016 without issues. Read and write latency on the DB's is good (2 - 5 ms).

    I saw another article online about garbage collection on IIS applications. When I check perfmon I can see that "MSExchangeServiceAppPool" has an average of 12% "% Time in Garbage Collection". I wonder if that could be an issue, as I could read that should be below 10%.

    Regards,
    Wouter


  3. Piotr Stanczak 1 Reputation point
    2022-03-16T10:55:08.383+00:00

    I have exactly the same issue on Exchange 2016 CU22, ~800 mailboxes most of them accessed in online mode.

    I am really keen to know if there is any progress on that issue.

    Regards
    Piotr


  4. Piotr Stanczak 1 Reputation point
    2022-03-16T12:56:48.853+00:00

    Hi Wouter,

    I have my own theory that the issue is related to online mode. I think that even if there is not a high load from users from the send/receive email perspective, every user click on the folder in Outlook or search is doing a lot of IIS connections because there is no cache. And I think this makes w3wp crash and after that Pool crashes.

    I also have ticket with MS will see what will come out from it. I will keep this thread updated if I have anything useful to share.
    Regards
    Piotr

    0 comments No comments

  5. Bitbear 1 Reputation point
    2022-03-17T00:28:33.777+00:00

    We are experiencing very similar issues on our Exchange On-prem server.

    Our environment is also Exchange 2019 server (CU10) running on Windows server 2019.
    Running with 8 cores avg around 15% usage, 96 GB memory avg 30%, 3 databases connected on an iSCSI disk avg 10% use time throughout the day. Approximately 300 mailboxes.

    We are having the same errors show up including System > WAS 5011 Warning , affecting what seems like all Exchange related app pools frequently at different times through the day, as well as Application > MSExchange Common 4999 Error

    Our users are reporting that Outlook is freezing/hanging or taking long times to load messages or search, as well as OWA becoming unavailable (not necessarily simultaneously).

    I've also checked the following with no further success:
    -run exchange health checker script > shows no problems
    -exclude exchange folders/processes/file types from anti-virus scanning based on Microsoft's recommendations

    I'm hoping we get some more responses here soon because I'm running out of ideas. I'll be sure to post if I make any progress though!