Start-MailboxAssistant (Exchange 2019 CU11) - the little command that promised so much and delivered.... nothing?

Question

Ever since upgrading to Exchange 2019 we have been plagued by search issues with BigFunnel - ref: here and here

I found this last week - https://tkolber.medium.com/fixing-search-indexing-issues-within-exchange-2019-mailboxes-eba04d207312

So over the weekend, I upgraded from CU10 to CU11 so that we could access that command. Finally completing the upgrade and after some sleep - i was eager to try this to see if all my problems would be behind me:

Except.... I don't think it does ANYTHING - so my question is (and probably someone from M$ needs to answer this one) - when the hell is it supposed to do and what should success look like?

Or better yet, Has anyone actually had success in using this command to lower the count of BigfunnelNotIndexedCount ??

Also for record - this command is NOT documented in the Exchange documentation on the MS site that I can find - nor is it found in the Github documentation repo

So lets pull this apart:

Starting with the help for the command, given that its not in the documentation:

Start-MailboxAssistant -?  
  
NAME  
    Start-MailboxAssistant  
  
SYNTAX  
    Start-MailboxAssistant [-Identity] <UserIdParameter> -AssistantName <string> [-Parameters <string>] [-SoftDeletedMailbox] [-DomainController <Fqdn>] [-ProxyToMailbox <MailboxIdParameter>] [-ProxyToServer <string>] [-WhatIf]  
    [-Confirm]  [<CommonParameters>]  
  
ALIASES  
    None  
  
REMARKS  
    None  
  

Right, nothing useful there about what should happen or what we should expect.

Well lets run it with Debug and verbose - maybe some logs will help

PS C:\Users\AdminExchange> Start-MailboxAssistant -Identity "REDACTED" -AssistantName BigFunnelRetryFeederTimeBasedAssistant -DomainController ActiveOpMasterADServer -Verbose -Debug  
VERBOSE: [00:48:57.742 GMT] Start-MailboxAssistant : Runspace context: Executing user: contoso.local/Accounts/contoso/Users/AdminExchange, Executing user organization: , Current organization: , RBAC-enabled: Disabled.  
VERBOSE: [00:48:57.745 GMT] Start-MailboxAssistant : Active Directory session settings for 'Start-MailboxAssistant' are: View Entire Forest: 'True',  
VERBOSE: [00:48:57.749 GMT] Start-MailboxAssistant : Beginning processing Start-MailboxAssistant  
VERBOSE: [00:48:57.752 GMT] Start-MailboxAssistant : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent".  
VERBOSE: [00:48:57.766 GMT] Start-MailboxAssistant : Current ScopeSet is: { Recipient Read Scope:{{, }}, Recipient Write Scopes:{{, }}, Configuration Read Scope:{{, }}, Configuration Write Scope(s):{{, }, }, Exclusive  
Recipient Scope(s):{}, Exclusive Configuration Scope(s):{} }  
VERBOSE: [00:48:57.776 GMT] Start-MailboxAssistant : Searching objects "REDACTED" of type "ADMailboxRecipient" under the root "$null".  
VERBOSE: [00:48:57.794 GMT] Start-MailboxAssistant : Previous operation run on domain controller 'ActiveOpMasterADServer.contoso.local'.  
VERBOSE: [00:48:57.797 GMT] Start-MailboxAssistant : The current object has been processed by the cmdlet extension agent with index 0.  
VERBOSE: [00:48:57.801 GMT] Start-MailboxAssistant : Processing object "contoso.local/Accounts/Redacted/Users/Redacted".  
VERBOSE: [00:48:57.805 GMT] Start-MailboxAssistant : Admin Audit Log: Entered Handler:Validate.  
VERBOSE: [00:48:57.809 GMT] Start-MailboxAssistant : Admin Audit Log: Entered ClassFactory:InitializeConfig.  
VERBOSE: [00:48:57.908 GMT] Start-MailboxAssistant : Admin Audit Log: Exited ClassFactory:InitializeConfig.  
VERBOSE: [00:48:57.965 GMT] Start-MailboxAssistant : Admin Audit Log: Exited Handler:Validate.  
VERBOSE: [00:48:57.967 GMT] Start-MailboxAssistant : Resolved current organization: .  
VERBOSE: [00:48:58.110 GMT] Start-MailboxAssistant : Invoking mailbox assistant for mailbox "contoso.local/Accounts/redacted/Users/redacted" via RPC call to remote server "RandomADServer.contoso.local".  
VERBOSE: [00:48:58.114 GMT] Start-MailboxAssistant : Searching objects "Redacted" of type "ADUser" under the root "$null".  
VERBOSE: [00:48:58.156 GMT] Start-MailboxAssistant : Previous operation run on domain controller 'ActiveOpMasterADServer.contoso.local'.  
VERBOSE: [00:48:58.160 GMT] Start-MailboxAssistant : Running assistant code for user: contoso.local/Accounts/Redacted/Users/Redacted  
VERBOSE: [00:48:58.177 GMT] Start-MailboxAssistant : Fetched mailbox session for user: redactedUUID  
VERBOSE: [00:48:58.182 GMT] Start-MailboxAssistant : Admin Audit Log: Entered Handler:OnComplete.  
VERBOSE: [00:48:58.186 GMT] Start-MailboxAssistant : Admin Audit Log: Exited Handler:OnComplete.  
VERBOSE: [00:48:58.188 GMT] Start-MailboxAssistant : Ending processing Start-MailboxAssistant  
PS C:\Users\AdminExchange>   

So it starts and finishes within 1.5 seconds.... right ok then.

Let's have a look behind the scenes at some logs to see what happens.....

#Fields: Timestamp,ServerName,AssistantName,Database,JobId,RequestType,MailboxGuid,Event,Reason,ExceptionType,InnerExceptionType,WorkcycleMinutes,AssistantCategory  
2022-02-07T01:06:33.723Z,EXSERVER213,BigFunnelRetryFeederTimeBasedAssistant,DB229,2022-02-07T01:06:32.9691377Z,OnDemand,REDACTEDUUID,StartProcessingMailbox,,,,240,Session  
2022-02-07T01:06:34.642Z,EXSERVER213,BigFunnelRetryFeederTimeBasedAssistant,DB229,2022-02-07T01:06:32.9691377Z,OnDemand,REDACTEDUUID,EndProcessingMailbox,,,,240,Session  

So.... it starts and finishes within a few seconds!? Pretty crazy given the number of partially / not / corrupt objects

BigFunnelIndexedCount                          : 182331  
BigFunnelPartiallyIndexedCount                 : 6820  
BigFunnelNotIndexedCount                       : 2657  
BigFunnelCorruptedCount                        : 12  
BigFunnelStaleCount                            : 17  
BigFunnelShouldNotBeIndexedCount               : 3440  
BigFunnelMailboxCreationVersion                : 0  

Ok, ill leave it alone and see what happens overnight and after the backup routine.......................
........................... the same - no change. Nada. Zip. Zilcho.

So, what the actual?

Moving mailboxes to other DBs isn't viable. The issues come back after (at most) a week. I'll be forever moving mailboxes around.
Over 2000 of my users already show symptoms of this problem. We need an inplace fix.

I'd really like to call on Microsoft here - this is a big problem a lot of people are having.

Answer 1

So I did some more testing today with various users.
After testing about 30 users - they all failed, but 5 distinct errors appeared in the logs across those users.

This also was over 30 different DBs and a DAGs worth of servers

I have no idea what this means and googling various parts of them doesn't really give me much info.
My read on this is that there may be multiple issues involved in this big funnel issue behind the scenes but.

It is important to note that all commands were run as a Domain and Exchange Org administrator - so permissions are ruled out effectively

Hopefully someone else can help me make heads or tales of this?
And @Microsoft - if you are listening - happy to provide unredacted versions ;)

2022-02-07T03:29:27.634Z,Failures,1,BigFunnelRetryFeederTimeBasedAssistant_34986803,,"Retry Failure for mailbox <redacted mailbox UUID> with error : DoRetryItem failed: Microsoft.Exchange.Data.Storage.ConversionFailedException: Content conversion: Body conversion failed. ---> Microsoft.Exchange.Data.TextConverters.TextConvertersException: data truncated\r\n   at Microsoft.Exchange.Data.TextConverters.Internal.RtfCompressed.RtfDecompressConverter.Run()\r\n   at Microsoft.Exchange.Data.TextConverters.ConverterStream.Read(Byte[] buffer, Int32 offset, Int32 count)\r\n   at Microsoft.Exchange.Data.TextConverters.RtfPreviewStream..ctor(Stream inputRtfStream, Int32 inputBufferSize)\r\n   at Microsoft.Exchange.Data.Storage.Body.<ChooseBestBody>b__120_0()\r\n   at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, LocalizedString exceptionString, CtsCall ctsCall)\r\n   --- End of inner exception stack trace ---\r\n   at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, LocalizedString exceptionString, CtsCall ctsCall)\r\n   at Microsoft.Exchange.Data.Storage.Body.ChooseBestBody()\r\n   at Microsoft.Exchange.Data.Storage.Body.get_Size()\r\n   at Microsoft.Exchange.Data.Storage.CalendarItemBase.CreateCacheForChangeHighlight()\r\n   at Microsoft.Exchange.Data.Storage.ItemCreateInfo.CalendarItemCreator(ICoreItem coreItem)\r\n   at Microsoft.Exchange.Data.Storage.ItemCreateInfo.Creator(ICoreItem coreItem, Boolean isCoreItemInitialized)\r\n   at Microsoft.Exchange.Data.Storage.ItemBuilder.ItemBind[T](IStoreSession session, StoreId storeId, Schema expectedSchema, MapiMessageCreator mapiMessageCreator, ItemBindOption itemBindOption, ICollection`1 propertiesToLoad, ItemCreateInfo itemCreateInfo)\r\n   at Microsoft.Exchange.Data.Storage.ItemRepository.Bind(StoreId storeId, ItemBindOption itemBindOption, ICollection`1 propsToReturn, Nullable`1 expectedStoreObjectType)\r\n   at Microsoft.Exchange.Data.Storage.Item.Bind(IStoreSession session, StoreId storeId, ItemBindOption itemBindOption, ICollection`1 propsToReturn)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.GetPreparedItem(IRetryContext retryContext)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.<>c__DisplayClass7_0.<Retry>b__0()\r\n   at Microsoft.Exchange.Data.Storage.MailboxSession.BypassAuditsFolderAccessChecking(Action action)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.Retry(IRetryContext retryContext)\r\n   at Microsoft.Exchange.MailboxAssistants.Assistants.BigFunnelRetryFeederTimeBasedAssistant.BigFunnelRetryFeederTimeBasedAssistant.DoRetryItem(BigFunnelRetryFeederTaskContext context)",  
  
2022-02-07T03:47:09.939Z,Failures,1,BigFunnelRetryFeederTimeBasedAssistant_52565294,,"Retry Failure for mailbox <redacted mailbox UUID> with error : RetriableItem.Retry for Item with StoreId: <redacted mailbox store ID>, BigFunnelCorrelationId: <redacted UUID>, Item Class:  failed with Exception: Microsoft.Exchange.Data.Storage.ConversionFailedException: Content conversion: Body conversion failed. ---> Microsoft.Exchange.Data.TextConverters.TextConvertersException: data truncated\r\n   at Microsoft.Exchange.Data.TextConverters.Internal.RtfCompressed.RtfDecompressConverter.Run()\r\n   at Microsoft.Exchange.Data.TextConverters.ConverterStream.Read(Byte[] buffer, Int32 offset, Int32 count)\r\n   at Microsoft.Exchange.Data.TextConverters.RtfPreviewStream..ctor(Stream inputRtfStream, Int32 inputBufferSize)\r\n   at Microsoft.Exchange.Data.Storage.Body.<ChooseBestBody>b__120_0()\r\n   at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, LocalizedString exceptionString, CtsCall ctsCall)\r\n   --- End of inner exception stack trace ---\r\n   at Microsoft.Exchange.Data.Storage.ConvertUtils.CallCts(Trace tracer, String methodName, LocalizedString exceptionString, CtsCall ctsCall)\r\n   at Microsoft.Exchange.Data.Storage.Body.ChooseBestBody()\r\n   at Microsoft.Exchange.Data.Storage.Body.get_Size()\r\n   at Microsoft.Exchange.Data.Storage.CalendarItemBase.CreateCacheForChangeHighlight()\r\n   at Microsoft.Exchange.Data.Storage.ItemCreateInfo.CalendarItemCreator(ICoreItem coreItem)\r\n   at Microsoft.Exchange.Data.Storage.ItemCreateInfo.Creator(ICoreItem coreItem, Boolean isCoreItemInitialized)\r\n   at Microsoft.Exchange.Data.Storage.ItemBuilder.ItemBind[T](IStoreSession session, StoreId storeId, Schema expectedSchema, MapiMessageCreator mapiMessageCreator, ItemBindOption itemBindOption, ICollection`1 propertiesToLoad, ItemCreateInfo itemCreateInfo)\r\n   at Microsoft.Exchange.Data.Storage.ItemRepository.Bind(StoreId storeId, ItemBindOption itemBindOption, ICollection`1 propsToReturn, Nullable`1 expectedStoreObjectType)\r\n   at Microsoft.Exchange.Data.Storage.Item.Bind(IStoreSession session, StoreId storeId, ItemBindOption itemBindOption, ICollection`1 propsToReturn)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.GetPreparedItem(IRetryContext retryContext)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.<>c__DisplayClass7_0.<Retry>b__0()\r\n   at Microsoft.Exchange.Data.Storage.MailboxSession.BypassAuditsFolderAccessChecking(Action action)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.Retry(IRetryContext retryContext)",  
  
2022-02-07T02:59:14.520Z,Failures,1,BigFunnelRetryFeederTimeBasedAssistant_58409158,,"Retry Failure for mailbox <redacted mailbox UUID> with error : DoRetryItem failed: Microsoft.Exchange.Data.Storage.AccessDeniedException: Cannot get attachment table. ---> Microsoft.Mapi.MapiExceptionNoAccess: MapiExceptionNoAccess: Unable to query table rows. (hr=0x80070005, ec=-2147024891)\nDiagnostic context:\n    Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=3855]\n    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=1065][latency=14]\n    Lid: 52176   ClientVersion: 15.2.986.5\n    Lid: 50032   ServerVersion: 15.2.986.6005\n    Lid: 35180  \n    Lid: 23226   --- ROP Parse Start ---\n    Lid: 27962   ROP: ropDeletePropsNoReplicate [122]\n    Lid: 17082   ROP Error: 0x80070005\n    Lid: 30561  \n    Lid: 21921   StoreEc: 0x80070005\n    Lid: 27962   ROP: ropExtendedError [250]\n    Lid: 1494    ---- Remote Context Beg ----\n    Lid: 52072   dwParam: 0x2\n    Lid: 60775   StoreEc: 0x80070005\n    Lid: 40748   qdwParam: 0xB01000000000001 \n    Lid: 57132   qdwParam: 0x7365287100000002\n    Lid: 63016   dwParam: 0x7A\n    Lid: 39640   StoreEc: 0x80070005\n    Lid: 45434   Guid: <redacted mailbox UUID>\n    Lid: 10786   dwParam: 0x0        Msg: 15.02.0986.005:<Redacted Server name> :<redacted UUID>\n    Lid: 1750    ---- Remote Context End ----\n    Lid: 27962   ROP: ropSetProps [10]\n    Lid: 26881  \n    Lid: 21817   ROP Failure: 0x80070005\n    Lid: 28414  \n    Lid: 32510   StoreEc: 0x80070005\r\n   at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)\r\n   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)\r\n   at Microsoft.Mapi.MapiTable.QueryRows(Int32 crows, QueryRowsFlags flags)\r\n   at Microsoft.Mapi.MapiTable.QueryAllRows(Restriction restriction, ICollection`1 propTags)\r\n   at Microsoft.Exchange.Data.Storage.MapiAttachmentProvider.QueryAttachmentTable(NativeStorePropertyDefinition[] properties)\r\n   --- End of inner exception stack trace ---\r\n   at Microsoft.Exchange.Data.Storage.MapiAttachmentProvider.QueryAttachmentTable(NativeStorePropertyDefinition[] properties)\r\n   at Microsoft.Exchange.Data.Storage.CoreAttachmentCollection.InternalOpen(AttachmentHandle handle, ICollection`1 preloadProperties)\r\n   at Microsoft.Exchange.Data.Storage.CoreAttachmentCollection.Open(IAttachmentHandle handle, ICollection`1 preloadProperties)\r\n   at Microsoft.Exchange.Data.Storage.Auditing.CoreItemExtensions.AttachmentsAsAuditString(ICoreItem coreItem)\r\n   at Microsoft.Exchange.Data.Storage.Adapters.ObjectEventHandlers.CopyOnWriteEventHandler.CacheItemInformation(CallbackContext cowContext, ICoreItem item, Boolean includeLegallyDirtyProperties)\r\n   at Microsoft.Exchange.Data.Storage.Adapters.ObjectEventHandlers.CopyOnWriteEventHandler.Handle(BeforeItemSaveEvent onBeforeItemSaveArgs)\r\n   at Microsoft.Exchange.Data.StoreObjects.Extensibility.InlineEvent.EventDispatcher.<>c__DisplayClass4_1`1.<Dispatch>b__0()\r\n   at Microsoft.Exchange.Data.StoreObjects.Extensibility.InlineEvent.EventDispatcher.Dispatch[TEvent](TEvent eventData)\r\n   at Microsoft.Exchange.Data.Storage.CoreItem.PreSave(OperationContext operationContext)\r\n   at Microsoft.Exchange.Data.Storage.CoreItem.InternalSave(SaveMode saveMode, OperationContext operationContext)\r\n   at Microsoft.Exchange.Data.Storage.CoreItem.SaveOrSend(SaveMode saveMode, OperationContext operationContext, CoreItemOperation operation)\r\n   at Microsoft.Exchange.Data.Storage.Item.SaveInternal(SaveMode saveMode, Boolean commit, OperationContext operationContext, CoreItemOperation operation)\r\n   at Microsoft.Exchange.Data.Storage.Item.Save(SaveMode saveMode)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.<>c__DisplayClass7_0.<Retry>b__0()\r\n   at Microsoft.Exchange.Data.Storage.MailboxSession.BypassAuditsFolderAccessChecking(Action action)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.Retry(IRetryContext retryContext)\r\n   at Microsoft.Exchange.MailboxAssistants.Assistants.BigFunnelRetryFeederTimeBasedAssistant.BigFunnelRetryFeederTimeBasedAssistant.DoRetryItem(BigFunnelRetryFeederTaskContext context)",  
  
2022-02-07T01:08:33.889Z,Failures,1,BigFunnelRetryFeederTimeBasedAssistant_15213362,,"Retry Failure for mailbox <redacted mailbox UUID> with error : RetriableItem.Retry for Item with StoreId: <redacted mailbox store ID>, BigFunnelCorrelationId: <redacted UUID>, Item Class: IPM.Note failed with Exception: Microsoft.Exchange.Data.Storage.MessageSubmissionExceededException: Cannot save changes made to an item to store. ---> Microsoft.Mapi.MapiExceptionMaxSubmissionExceeded: MapiExceptionMaxSubmissionExceeded: Unable to save changes. (hr=0x80004005, ec=1242)\nDiagnostic context:\n    Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=133]\n    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=380][latency=1]\n    Lid: 52176   ClientVersion: 15.2.986.5\n    Lid: 50032   ServerVersion: 15.2.986.6005\n    Lid: 35180  \n    Lid: 23226   --- ROP Parse Start ---\n    Lid: 27962   ROP: ropSaveChangesMessage [12]\n    Lid: 17082   ROP Error: 0x4DA     \n    Lid: 18273  \n    Lid: 21921   StoreEc: 0x4DA     \n    Lid: 27962   ROP: ropExtendedError [250]\n    Lid: 1494    ---- Remote Context Beg ----\n    Lid: 59176   dwParam: 0x3200000  Msg: Limitation\n    Lid: 34600   dwParam: 0x4A5AF12  Msg: CurrentSize\n    Lid: 49178   dwParam: 0x0\n    Lid: 42792   dwParam: 0x0\n    Lid: 36890   StoreEc: 0x4DA     \n    Lid: 40748   qdwParam: 0x647E780000000001\n    Lid: 57132   qdwParam: 0x9BEA7A0000000001\n    Lid: 63016   dwParam: 0xC\n    Lid: 39640   StoreEc: 0x4DA     \n    Lid: 45434   Guid: <redacted mailbox UUID>\n    Lid: 10786   dwParam: 0x0        Msg: 15.02.0986.005:<redacted server name>:<redacted UUID>\n    Lid: 1750    ---- Remote Context End ----\n    Lid: 31418   --- ROP Parse Done ---\n    Lid: 21457  \n    Lid: 19665   StoreEc: 0x4DA     \n    Lid: 37632  \n    Lid: 37888   StoreEc: 0x4DA     \r\n   at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)\r\n   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)\r\n   at Microsoft.Mapi.MapiProp.SaveChanges(SaveChangesFlags flags)\r\n   at Microsoft.Exchange.Data.Storage.MapiPropertyBag.SaveChanges(Boolean force)\r\n   --- End of inner exception stack trace ---\r\n   at Microsoft.Exchange.Data.Storage.MapiPropertyBag.SaveChanges(Boolean force)\r\n   at Microsoft.Exchange.Data.Storage.StoreObjectPropertyBag.SaveChanges(Boolean force)\r\n   at Microsoft.Exchange.Data.Storage.AcrPropertyBag.SaveChanges(Boolean force)\r\n   at Microsoft.Exchange.Data.Storage.CoreItem.InternalSave(SaveMode saveMode, OperationContext operationContext)\r\n   at Microsoft.Exchange.Data.Storage.CoreItem.SaveOrSend(SaveMode saveMode, OperationContext operationContext, CoreItemOperation operation)\r\n   at Microsoft.Exchange.Data.Storage.Item.SaveInternal(SaveMode saveMode, Boolean commit, OperationContext operationContext, CoreItemOperation operation)\r\n   at Microsoft.Exchange.Data.Storage.Item.Save(SaveMode saveMode)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.<>c__DisplayClass7_0.<Retry>b__0()\r\n   at Microsoft.Exchange.Data.Storage.MailboxSession.BypassAuditsFolderAccessChecking(Action action)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.Retry(IRetryContext retryContext)",  
  
2022-02-07T07:47:15.130Z,Failures,1,BigFunnelRetryFeederTimeBasedAssistant_52565294,,"Retry Failure for mailbox <redacted mailbox UUID> with error : RetriableItem.Retry for Item with StoreId: <redacted mailbox store ID>, BigFunnelCorrelationId: <redacted UUID>, Item Class:  failed with Exception: Microsoft.Exchange.Data.Storage.ObjectNotFoundException: The process failed to get the correct properties. ---> Microsoft.Mapi.MapiExceptionNotFound: MapiExceptionNotFound: Unable to get properties on object. (hr=0x8004010f, ec=-2147221233)\nDiagnostic context:\n    Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=1159]\n    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=290][latency=0]\n    Lid: 52176   ClientVersion: 15.2.986.5\n    Lid: 50032   ServerVersion: 15.2.986.6005\n    Lid: 35180  \n    Lid: 23226   --- ROP Parse Start ---\n    Lid: 27962   ROP: ropOpenMessage [3]\n    Lid: 17082   ROP Error: 0x8004010F\n    Lid: 26977  \n    Lid: 21921   StoreEc: 0x8004010F\n    Lid: 27962   ROP: ropExtendedError [250]\n    Lid: 1494    ---- Remote Context Beg ----\n    Lid: 50428  \n    Lid: 59416   StoreEc: 0x8004010F\n    Lid: 38536  \n    Lid: 45434   Guid: <redacted UUID>\n    Lid: 10786   dwParam: 0x0        Msg: 15.02.0986.005:<redacted Server name>:<redacted UUID>\n    Lid: 1750    ---- Remote Context End ----\n    Lid: 27962   ROP: ropGetPropsSpecific [7]\n    Lid: 17082   ROP Error: 0x4B9     \n    Lid: 26465  \n    Lid: 21921   StoreEc: 0x4B9     \n    Lid: 31418   --- ROP Parse Done ---\n    Lid: 22753  \n    Lid: 21817   ROP Failure: 0x4B9     \n    Lid: 20385  \n    Lid: 28577   StoreEc: 0x8004010F\n    Lid: 32001  \n    Lid: 29953   StoreEc: 0x8004010F\n    Lid: 32768  \n    Lid: 33024   StoreEc: 0x8004010F\r\n   at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)\r\n   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)\r\n   at Microsoft.Mapi.MapiProp.GetProps(ICollection`1 propTagsRequested)\r\n   at Microsoft.Exchange.Data.Storage.MapiPropertyBag.GetProperties(IList`1 propertyDefinitions)\r\n   --- End of inner exception stack trace ---\r\n   at Microsoft.Exchange.Data.Storage.MapiPropertyBag.GetProperties(IList`1 propertyDefinitions)\r\n   at Microsoft.Exchange.Data.Storage.StoreObjectPropertyBag.InternalLoad(ICollection`1 extraProperties)\r\n   at Microsoft.Exchange.Data.Storage.StoreObjectPropertyBag.Load(ICollection`1 properties)\r\n   at Microsoft.Exchange.Data.Storage.StoreObjectPropertyBag..ctor(IStoreSession session, MapiProp mapiProp, ICollection`1 autoloadProperties, Boolean canSaveOrDisposeMapiProp)\r\n   at Microsoft.Exchange.Data.Storage.CoreItem.CoreItemFactory.CoreItemBind(IStoreSession session, StoreId storeId, MapiMessageCreator mapiMessageCreator, ItemBindOption itemBindOption, ICollection`1 propertiesToLoad, StoreObjectType& storeObjectType)\r\n   at Microsoft.Exchange.Data.Storage.ItemBuilder.ItemBind[T](IStoreSession session, StoreId storeId, Schema expectedSchema, MapiMessageCreator mapiMessageCreator, ItemBindOption itemBindOption, ICollection`1 propertiesToLoad, ItemCreateInfo itemCreateInfo)\r\n   at Microsoft.Exchange.Data.Storage.ItemRepository.Bind(StoreId storeId, ItemBindOption itemBindOption, ICollection`1 propsToReturn, Nullable`1 expectedStoreObjectType)\r\n   at Microsoft.Exchange.Data.Storage.Item.Bind(IStoreSession session, StoreId storeId, ItemBindOption itemBindOption, ICollection`1 propsToReturn)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.GetPreparedItem(IRetryContext retryContext)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.<>c__DisplayClass7_0.<Retry>b__0()\r\n   at Microsoft.Exchange.Data.Storage.MailboxSession.BypassAuditsFolderAccessChecking(Action action)\r\n   at Microsoft.Exchange.Search.BigFunnel.RetryFeeder.RetriableItem.Retry(IRetryContext retryContext)",  


  

Answer 2

Hello,

I am facing the same problem as you.. and I have been looking for a viable solution for days but without success. Have you made any progress on the subject?

Answer 3

Hi guys,

I had the same issue with Exchange 2019 CU12 SU23.

What solved the problem were both sommands:

New-SettingOverride -Name "Big Funnel Permanent Retry" -Component "BigFunnel" -Section "BigFunnelFailedItemsQuerySettings" -Parameters @("ExcludePermanentFailures=false") -Reason "Re-Index Messages" -MinVersion "15.02.1118.21" -MaxVersion "15.02.1258.012"

and

Start-MailboxAssistant -Identity "Affected_User" -AssistantName BigFunnelRetryFeederTimeBasedAssistant -DomainController PrimaryDomainController -Verbose -Debug

for each of the affected mailboxes

And verify until following command doesn't return any results:

Get-Mailbox | Get-MailboxStatistics | ? {$_.BigfunnelNotIndexedCount -ge "1"} | ft DisplayName,BigfunnelNotIndexedCount