In Exchange Server 2013 Exchange Org Admin / Recipient Admin cannot modify USG Ownership(Managedby) / Membership of mail enabled Universal security groups that they do not OWN in Exchange admin center (EAC) . following Error is logged If try to modify members or Owners ,

 " You don't have sufficient permissions. This operation can only be performed by a manager of the group."

This behavior is by design in Exchange Server 2013 ( EAC), while modifying Security Group Owners/membership that they do not own. Only owners of a security group can add members to the Security Group or remove members from the security group using ECP/EAC.

Add-ADPermission pF01 -User user01 -ExtendedRights send-as

Active Directory operation failed on DC01.Corp.M16.com. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
 + CategoryInfo          : WriteError: (0:Int32) [Add-ADPermission], ADOperationException
 + FullyQualifiedErrorId : B3EE6A10,Microsoft.Exchange.Management.RecipientTasks.AddADPermission

When Public folder is created From a specific Server, Only the specific Exchange 2010 PF server will have the permissions to modify the "Send-As" Rights, As that server is the Owner of the Ad Object that corresponds to mail-enabled Public folder. When run the Add-ADPermission cmdlet to manage 'Send-As' permission on public folder from the Other Exchange 2010 server Other than where PF was created, Exchange will be access denied to modify the permissions on mail-enabled PF Object In MESO.

- Additionally you can verify AD Permissions using DSACLS , ADPermission or windows PowerShell.

When manage "Send-as" request is sent using Exchange Management shell or GUI , Scope are verified and validated before presenting the credentials of the user for modification ,

"ADSession::IsWithinScope 'CN=PF01,CN=Microsoft Exchange System Objects,DC=Corp,DC=M16,DC=com' is within scope. ScopeRoot '<null>', ScopeFilter '(!((Exists(ConfigurationUnit))))'"
"ADSession::IsWithinScope 'CN=PF01,CN=Microsoft Exchange System Objects,DC=Corp,DC=M16,DC=com' is within scope. ScopeRoot '<null>', ScopeFilter '<null>'"
"ADSession::IsWithinScope 'CN=PF01,CN=Microsoft Exchange System Objects,DC=Corp,DC=M16,DC=com' is within scope. ScopeRoot '<null>', ScopeFilter '(!((Exists(ConfigurationUnit))))'"
"ADSession::IsWithinScope 'CN=PF01,CN=Microsoft Exchange System Objects,DC=Corp,DC=M16,DC=com' is within scope. ScopeRoot '<null>', ScopeFilter '<null>'"
"GetConnection","Returning connection to DC01.Corp.M16.com:389"
"ADSession::ExecuteModificationRequest using DC01.Corp.M16.com:389 - Sending ModifyRequest request for CN=PF01,CN=Microsoft Exchange System Objects,DC=Corp,DC=M16,DC=com"
"DirectoryException","Caught System.DirectoryServices.Protocols.DirectoryOperationException with 50(0x32), message=00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0" 

 This is generic ACCESS Denied Error.  

Currently The Exchange Trusted Subsystem (ETS) is not granted sufficient rights to create a Manage "Send-As" permissions on Publicfolder Objects in MESO Container in AD. Currently ETS can only manage the Send-as Permissions for these objects.
== This is a problem when you have several public folder servers and many people are allowed to create public folders, and if you are managing "Send-as" permission on public folder objects in AD. because you cannot manage them from every server, other than where they are created.

There are couple workarounds to fix :

A : It is very manual method, If you use ADSIEdit to change the Owner of the PF object in AD to be Exchange 2010 server B or C or D , then you can grant Send As permissions from server B OR C or D. but not from server A anymore.

so not an easy fix.

B : Assign Permissions on MESO Container for ETS ( Exchange trusted Subsystem) to "Modify Permission" . [ Similar to /Preparead ]

Open ADSIEDIT =] Navigate to the properties of the MESO ( Microsoft Exchange System Objects) container -- Select "Security"  tab ---> Select “advanced” Tab at the bottom " ---> In the Add Permissions window Select  “add” button ---> Add “Exchange trusted subsystem”

And assign “Modify permissions”. Permission

Select “This object and all decedent objects"

==  Now "Send-as" Permissions for Mail enabled public folders can be managed from any Exchange 2010 server in the Organization.

Manju

[Troubleshooting]- Content Index 'status' shows "Failed" on all mailbox databases.

[Troubleshooting]- Update 2874216 (MS13-061) breaks the content index in Exchange Server 2013.

[Blog/Troubleshooting]- Exchange 2013 Upgrade/Uninstall fails while uninstall Search Foundation for Exchange Server 2013

[Blog/Info]- Exchange 2013 Fast Search, HA and the Windows Registry

[Troubleshooting] - Content Index state is 'Failed' and ‘AdminNodes’ are terminating

[Troubleshooting/kb] - Exchange Server 2013 Setup fails when Configuring Search Foundation - [Sept-2013]

[Troubleshooting/kb - Exchange 2013 Search Failed on All databases ( NodeRunner.exe terminated) - [Apr-2014]

Manju

Microsoft Exchange Host Controller Service starts 4 worker processes, each named NodeRunner (Admin Nodes). The Individual functionality of Each NodeRunner process is conferred by  configuration. The NodeRunner.exe  which activates single Admin node in a process of its own. NodeRunner.exe is a standalone process, it derives some of the operating properties from  NodeRunner.exe.config the application configuration file during the start of Admin Nodes.

* if you are familiar with SPS, you probably know how to Limit the Memory allocated to NodeRunner.exe Process using the application configuration file. ( Do Not implement SPS resolutions/ workaround in Exchange  server 2013).

The default setting for minimum memory requirement for NodeRunner.exe  is 0. NodeRunner process can consume and change its memory requirements dynamically based on the requirement and available memory. However, you can set the Upper limit and restrict the amount of memory that node runner can access using the parameter "memoryLimitMegabytes" in the NodeRunner.exe.config application Configuration file. ( **** Lessons from SPS...  ). If you limit the memory usage by NodeRunner.exe and if we can't allocate the memory for NodeRunner.exe  operation, we fail with an OutOfMemoryException. Check for the following events if you are experiencing this issue and for a resolution.

[ * limiting memory allocation for NodeRunner.exe in Exchange server 2013 using this method is not supported ].

Symptoms :

A - Content Index stats on all mailbox databases are "FailedAndSuspended"

B - ULS Logs :    

 NodeRunnerIndexNode1-Fsis (0x59D4) 0x6390 Search  Search Component  
 IndexRouter[AA51A323-AFF5-490F-BEC8-40300AB578BC12]: Caught exception when preparing generation GID[38]: (IndexNode1-00000000-0000-0000-0000-000000000000-AA51A323-AFF5-490F-BEC8-40300AB578BC12.Single: bad allocation): Microsoft.Ceres.SearchCore.FastServer.FastServerException:
IndexNode1-00000000-0000-0000-0000-000000000000-AA51A323-AFF5-490F-BEC8-40300AB578BC12.Single: bad allocation   
at Microsoft.Ceres.SearchCore.FastServer.Indexer.PrepareGeneration()    
at Microsoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.FlushGeneration()    
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractProtectedGenerationBuilder.FlushGenerationInternal()    
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractBuilder.PrepareCloseGeneration()    
at Microsoft.Ceres.SearchCor... 

c - Event Logs:

The NodeRunner.exe process throws an exception when trying to allocate more memory than the limited memory.                                                                                                                    

ID:1026
Level:Error
Source:.NETRuntime
Message: Application: NodeRunner.exe
Framework Version:v4.0.30319
Description: The process was terminated due to an unhandled exception
Exception Info: System. OutOfMemoryException
Stack:
atMicrosoft.Ceres.CoreServices.Framework.Container.ComponentManager.FailingInvoker()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback,System.Object)
at System.Threading.ThreadHelper.ThreadStart() 

ID:1000
Source: Application
Error Message:
Faulting application name: NodeRunner.exe,
version: 16.0.847.30,
time stamp: 0x52fef8d5
Faulting module name: ntdll.dll,
version: 6.2.9200.16579,
time stamp: 0x51637f77
Exception code: 0xc00000fd
Fault offset:
0x000000000001b45c
Faulting process id:0x12b0
Faulting application start time:0x01cf4e43dce6bead
Faulting application path: C:\Program Files\Microsoft\ExchangeServer\V15\Bin\Search\Ceres\Runtime\1.0\NodeRunner.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id:29ffb9d1-ba37-11e3-9427-00155d0a730c
Faulting package full name: Faulting package-relative application ID: 

ID:  1026
Level:Error
Source: .NET Runtime
Application: NodeRunner.exe
Framework Version:v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info:
System.AccessViolationException
Stack:  
atMicrosoft.Ceres.SearchCore.FastServer.Indexer.PrepareGeneration()
atMicrosoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.FlushGeneration()
atMicrosoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.PublishGeneration()
atMicrosoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractBuilder.CloseGeneration()                                                                     atMicrosoft.Ceres.SearchCore.Journal.RegistrationHandler.CloseGenerationToUsers(Microsoft.Ceres.SearchCore.Journal.RecoveryState)
atMicrosoft.Ceres.SearchCore.Journal.Journal.RecoveryEndGeneration(Microsoft.Ceres.SearchCore.Services.GenerationId)
at Microsoft.Ceres.SearchCore.Journal.RegistrationHandler.RedoJournalEntry(Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry, UInt16,Microsoft.Ceres.SearchCore.Journal.RecoveryState)
at Microsoft.Ceres.SearchCore.Journal.Journal.RecoverEntry(Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry, UInt16)
at Microsoft.Ceres.SearchCore.Journal.DoubleBuffered.DoubleBufferJournal.RecoveryScan(Microsoft.Ceres.SearchCore.Services.Journal.CheckpointId, Microsoft.Ceres.SearchCore.Journal.JournalEntryRecovered, Microsoft.Ceres.SearchCore.Journal.ResumeCheckpointing, System.Collections.Generic.IDictionary`2<UInt16,Microsoft.Ceres.Common.Utils.IFactory`1<Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry>>)      atMicrosoft.Ceres.SearchCore.Journal.Journal.DoLogicalRecovery()
atMicrosoft.Ceres.SearchCore.Journal.Journal.EnsureFullRecovery()
atMicrosoft.Ceres.SearchCore.Journal.JournalComponent.DoReadying()
atSystem.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)  atSystem.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

ID:       1000
Level:    Error
Message:
Faulting application name: NodeRunner.exe, version: 16.0.847.30, time stamp: 0x52fef8d5
Faulting module name: MSVCR110.dll, version: 11.0.51106.1, time stamp: 0x5098826e
Exception code: 0xc0000005 Fault offset: 0x000000000003c920
Faulting process id: 0x6670
Faulting application start time: 0x01cf4e437e35974d
Faulting application path: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Runtime\1.0\NodeRunner.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR110.dll Report Id: cc6806ff-ba36-11e3-9427-00155d0a730c Faulting package full name: Faulting package-relative application ID: 

Fix NodeRunner Exception issue :

Events in ULS logs and event logs indicate NodeRunners.exe are terminating with exception, because of  OutOfMemory. Either the server is outofmemory or .Net CLR is forcing memory allocation limitation to the NodeRunners.exe process. NodeRunner tried to allocate more memory and terminates with an exception .
Navigate to following location , edit the file  

C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config

Check for Key and the value  <nodeRunnerSettings memoryLimitMegabytes="0" />.

if it is set to any value than 0, change it to 0 and restart  "Microsoft Exchange Host Controller Service"

_{"An unexpected error occurred and your request couldn't be handled"}

_Request

_{Url: https://localhost:443/owa/ev.owa?oeh=1&ns=MsgVLV2&ev=LoadFresh&pfmk=M23:1390268685136}

_{User: John Doe}

_{EX Address: /o=Contoso/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=John Does7314f2d}

_{SMTP Address: JohnDoe@contoso.com}

_{Mailbox server: <FQDN OF Mailbox SERVER Generating Exception >}

_Exception

_{Exception type: System.TypeInitializationException}

Exception message: The type initializer for 'Microsoft.Exchange.Data.Search.AqsParser.AqsParse' threw an exception.

 _{Call stack}r' threw an exception.

 _{Call stack}

 _{Microsoft.Exchange.Data.Search.AqsParser.AqsParser.ParseAndBuildQuery(String query, ParseOption parseOption, CultureInfo culture, Boolean isContentIndexingEnabled, RescopedAll rescopedAll, IRecipientResolver recipientResolver, IPolicyTagProvider policyTagProvider)}

_{Microsoft.Exchange.Clients.Owa.Core.SearchFilterGenerator.Execute(String searchString, Boolean isContentIndexingEnabled, Folder folder, SearchScope searchScope)}

_{Microsoft.Exchange.Clients.Owa.Core.SearchFilterGenerator.Execute(String searchString, Boolean isContentIndexingEnabled, CultureInfo userCultureInfo, IPolicyTagProvider policyTagProvider, Folder folder, SearchScope searchScope, QueryFilter advancedQueryFilter)}

_{Microsoft.Exchange.Clients.Owa.Core.FolderSearch.Execute(UserContext userContext, Folder folder, SearchScope searchScope, String searchString, Boolean newSearch, Boolean asyncSearch)}

_{Microsoft.Exchange.Clients.Owa.Premium.FolderVirtualListViewEventHandler2.BindToFolder()}

_{Microsoft.Exchange.Clients.Owa.Premium.MessageVirtualListViewEventHandler2.GetListView()}

_{Microsoft.Exchange.Clients.Owa.Premium.VirtualListViewEventHandler2.InternalLoadFresh(Boolean renderHeaders)}

_{Microsoft.Exchange.Clients.Owa.Premium.MessageVirtualListViewEventHandler2.LoadFresh()}

 _{Inner Exception}

_{Exception type: Microsoft.StructuredQuery.StructuredQueryException}

_{Exception message: Exception of type 'Microsoft.StructuredQuery.StructuredQueryException' was thrown.}

 _{Call stack}

 _{Microsoft.StructuredQuery.Parser.Parse(String strQuery, List`1 lstCustomProperties)}

_{Microsoft.Exchange.Data.Search.AqsParser.AqsParser..cctor()}

_{==> Additonally you may see these events in event logs}

_{Event id: 9877}

_{Source:MSExchangeIS Mailbox Store}

_{Task Category: Content Indexing}

_{Description: Content Indexing function 'CISearch::EcGetRowsetAndAccessor' received an unusual and unexpected error code from MSSearch. ....Error Code: 0x80070781.}

_{Event id : 1334}

_{Application Source:ASP.NET 2.0.50727.0}

_{Description: An unhandled exception occurred and the process was terminated}

_{Event ID:4999}

_Description:

_{Watson report about to be sent for process id: 14240, with parameters: E12, c-RTL-AMD64, 14.xx.xxxx.000, AirSync, M.Exchange.StructuredQuery, M.S.Parser.Parse,}

_{What causes this issue?}

_{This problem typically is a result of Missing Critical Operating system files which is responsible for transferring queries from OWA virtual directories to mailbox server search indexes.}

_{Reviewing the Servicing logs on Mailbox server, you could find missing system files,}

_{There could be several hundreds of them, from “\systemroot\winsxs\...” folder and “c:\windows\system32….”   folder.}

 _{[Error,Facility=(system),Code=640 (0x0280)] #3186416# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowFileNotFound|AllowSharingViolation|AllowAccessDenied), handle = {provider=NULL, handle=0}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA), oa = @0xacd1d0->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[129]"\SystemRoot\WinSxS\amd64_microsoft-windows-c..ent-indexing-common_31bf3856ad364e35_6.1.7601.17514_none_64da1339edafdc37\Query.dll"; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0xacd280, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_NON_DIRECTORY_FILE|FILE_SYNCHRONOUS_IO_NONALERT), eab = NULL, eal = 0, disp = Invalid)[gle=0xd0000280]}

_{[Error,Facility=(system),Code=640 (0x0280)] #3186414# from Windows::Rtl::SystemImplementation::CDirectory_IRtlDirectoryTearoff::OpenExistingFile(flags = (MissingFileIsOk|SharingViolationIsOk|AccessDeniedIsOk), da = (SYNCHRONIZE|FILE_READ_DATA), oa = @0xacd878->SIL_OBJECT_ATTRIBUTES {s:40; on:"Query.dll"; a:(OBJ_CASE_INSENSITIVE)}, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), oo = (FILE_SYNCHRONOUS_IO_NONALERT|FILE_NON_DIRECTORY_FILE), file = NULL, disp = Invalid)[gle=0xd0000280]}

_{[Error,Facility=(system),Code=640 (0x0280)] #3318672# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowFileNotFound|AllowSharingViolation|AllowAccessDenied), handle = {provider=NULL, handle=0}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA), oa = @0xacd1d0->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[127]"\SystemRoot\WinSxS\amd64_microsoft-windows-content-filter-html_31bf3856ad364e35_7.0.7600.16385_none_a7c20a5fb80f0e97\nlhtml.dll"; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0xacd280, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_NON_DIRECTORY_FILE|FILE_SYNCHRONOUS_IO_NONALERT), eab = NULL, eal = 0, disp = Invalid)[gle=0xd0000280]}

_{[Error,Facility=(system),Code=640 (0x0280)] #3318672# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowFileNotFound|AllowSharingViolation|AllowAccessDenied), handle = {provider=NULL, handle=0}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA), oa = @0xacd1d0->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[127]"\SystemRoot\WinSxS\amd64_microsoft-windows-content-filter-html_31bf3856ad364e35_7.0.7600.16385_none_a7c20a5fb80f0e97\nlhtml.dll"; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0xacd280, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_NON_DIRECTORY_FILE|FILE_SYNCHRONOUS_IO_NONALERT), eab = NULL, eal = 0, disp = Invalid)[gle=0xd0000280]}

 _{[Error,Facility=(system),Code=640 (0x0280)] #4047227# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowFileNotFound|AllowSharingViolation|AllowAccessDenied), handle = {provider=NULL, handle=0}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA), oa = @0xacd1d0->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[129]"\SystemRoot\WinSxS\amd64_microsoft-windows-i..nal-nlsdownleveldll_31bf3856ad364e35_6.1.7600.16385_none_649df50371b42c21\Nlsdl.dll"; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0xacd280, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_NON_DIRECTORY_FILE|FILE_SYNCHRONOUS_IO_NONALERT), eab = NULL, eal = 0, disp = Invalid)[gle=0xd0000280]}

_{<snip …. >}

_{Here is an example of corrupted / Missing system files,}

How do we resolve this issue?

- You Know the Answer :)

- Manju

A. Test if MSearch is experiencing deadlock situation, Run Troubleshoot-CI.PS1 Script from Exchange Management shell. The script checks the state of Exchange content index service.

 Troubleshoot-CI.ps1 -Server “Mailbox Server name”

 IsDeadLocked        : False

 IsRTMServer          : False

 IsMSFTEdMemoryConsumptionOverLimit: False

 * If deadlock Situation is detected follow This TechNet article to help resolve the issue:   Indexing backlog detected

 B. Run Get-mailboxdatabasecopystatus cmdlet

     Verify the content index status for all databases, check if all the DB’s are affected or only the few  

     If ContentIndexState == Failed follow the steps to address the issue: Reseed the Search Catalog

     ContentIndexState == Unknown, 

 **    In this case, Reset search index will not help, because the search state is unknown.

Error UnknownError (0x80043629) processing restriction by CI

CiSearch::EcGetRowsetAndAccessor(): ………… UnknownError (0x80043629)

0x80043629 -- is FTE_E_CATALOG_DOES_NOT_EXIST. Content indexing service gets this error from MSfteSql when the catalog was not found or was dismounted

When reviewing the Application log the following events are present

 ID: 9877
Level: Error
Source: MSExchangeIS Mailbox Store
Message: Content Indexing function 'CISearch::EcGetRowsetAndAccessor' received an unusual and unexpected error code from MSSearch

-- Restart Microsoft Exchange Search Service and verify if an issue is fixed.

-- Check if SYM Links are uninstalled

There is a symbolic link for each registry entry in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ContentIndex\Language  

Following script helps to rebuild broken Sym links.    Repair broken Sym links

** Above steps may not help occasionally and one need to troubleshoot further to isolate the issue   

C. Extrace to identify the issue with MSSearch,

MDB .. DB GUID…. - Adding document 163671 3e0c-2A113C-4-5CC72 to failed documents with error 0x80040d16 (UnknownError (0x80040d16))

MDB .. DB GUID…. - Adding document 163685 3e0c-2A113D-4-AFE52 to failed documents with error 0x80040d16 (UnknownError (0x80040d16))

- Errors listed above indicates the issue is with the corrupt search indexer files. Rebuilding catalog or restarting search index service or re-installing filter packs do not help. Follow the steps below to fix the issue.

-- Remove all installed filter packs including any third party filter packs 

-- Restart the server

-- Install Office Filter pack + SP for office filter pack if any

* Check that the search is working and returning results in OWA and Outlook (Online) 

 **If you have databases of large size or over Million items, you might experience a situation where you will have to index one database at a time.           

 Manju

However, Get-MailboxdatabaseCopyStatus reported "Failed" state on all Mailbox databases.

- Restart "HostControllerService".
  Service restarted, but AdminNodes failed to start  with HostCotrollerService.  

- Resetting the Content index on every database do not help, because Admin nodes are not running.
  Index won't be generated if AdminNodes cannot talk to each other.

Following Exception event is logged in event logs,

 An operation attempted against a FAST endpoint Experienced an exception. This Operation may be
retried. Error details:Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An
Exception was received during a FAST operation.--> System.ServiceModel.EndpointNotFoundException: Could
not connect to net.tcp://localhost:3803/Management/ContentTransformation/FlowService.The connect
connection attempt lasted for a time span of 00:00:02.0468713. TCP error code 10061: No Connection could be
made because the Target actively refused it 127.0.0.1:3803.  --->System.Net.Sockets.SocketException:
No connection could be made because the target machine actively refused it 127.0.0.1:3803 at
System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)   at
System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)

   --- End of inner exception stack trace ------------------------------------------------------------------------------

There are many methods to identify the cause, ULS Logging, 

 hostcontrollerservice.exe (0x3B94) - 0x5F0C -HostController  - HostController: Restarting node AdminNode1 on system
Fsis marked for automatic startup - threadId: 3 

hostcontrollerservice.exe (0x3B94) - 0x5F0C - HostController - HostController: Restarting node ContentEngineNode1 on
System Fsis marked for automatic startup - threadId: 3 

hostcontrollerservice.exe (0x3B94)- 0x5F0C - HostController - HostController: Restarting node IndexNode1 on System
Fsis marked for automatic startup - threadId: 3

hostcontrollerservice.exe (0x3B94) - 0x5F0C- HostController - HostController: Restarting node InteractionEngineNode1
on system Fsis marked for automatic startup - threadId:3 

NodeRunner.exe (0x5D94) - 0x58C4 - ULS Init -Completed NodeRunner.exe,Microsoft.Ceres.Diagnostics.Native.dll) 

Each node runner terminates with following error:

NodeRunnerIndexNode1-Fsis (0x1688) - Search Platform Services   - Unexpected exception in node
activator:  System.ArgumentException: An item with the same key has already been added. at
System.Collections.Generic.Dictionary`2.Insert(TKeykey, TValue value, Boolean add) at
Microsoft.Ceres.CoreServices.Node.BootstrapPropertyLoader.AddEnvironmentProperties(IDictionary`2
result) at Microsoft.Ceres.CoreServices.Node.NodeActivator.InitializeBootstrapProperties(IDictionary`2
overlay) at Microsoft.Ceres.CoreServices.Node.NodeActivator.ActivateNode(IDictionary`2 configuration) 

 In general Error would indicate that ‘AdminNodes’ are trying to add a Directory list and it is
duplicated. The duplicated entry is not same as the existing list.
To list Environment variable Properties via PS cmdlet

Get-ChildItem Env:

Look for multiple PSModulepath entries listed in Environment Variable  or PSModulepath entry with trailing space

    PSModulePath=%Unders%\Administrator.Domain\WindowsPowerShell\Modules;c:\Windows\System32\WindowsPowerShell\v1.0\Modules

   PSModulePath  =c:\Program Files\MicrosoftSecurity Client\MpProvider

                       [* Note a Trailing Space in " PSmodulepath  = " ]

In Our case Second PSModulePath entry was added by installing the Endpoint Protection Agent that
created an additional PSModulePath environment variable that contains a trailing space.

To fix the issue:

1. Install https://support.microsoft.com/kb/2865173
 if one of the following applications are installed on Exchange 2013 / Windows 2012 server 

• Microsoft System Center 2012 Configuration
  Manager Service Pack 1
• Microsoft Forefront Endpoint Protection 2010
• Microsoft System Center Configuration Manager 2007

2. Remove any trailing space from Environment variable “ PSModulePath  = "  and restart HostControllerService. 

    PSModulePath=c:\ProgramFiles\MicrosoftSecurity Client\MpProvider  [ <----Correct Entry ]

3. Fix the first entry by adding MPdirectory and Delete the Second one

PSModulePath=%Unders%\Administrator.Domain\WindowsPowerShell\Modules;c:\Windows\System32\
WindowsPowerShell\v1.0\Modules;c:\ProgramFiles\MicrosoftSecurity Client\MpProvider   

 - Restart HostConterllerService and Microsoft Exchange Search Service.

4. If one Entry for PSModulePath Remove the Trailing space in [ PSModulePath= "  ]  Restart HostControllerService.

 Manju

Deploying RU4 or later Rollup updates or Services packs Enabled Support for processing Calendar and Tasks. If you are already using retention policy, Prior to RU4 , Default Retention policy (DPT) cannot process Calendar and Tasks folder. Deploying RU4 or later service packs(SP3..) or Later RU automatically starts processing Calendar and tasks folder. Calendar and Tasks Folder will Inherit the default policy tag(DPT) automatically.

The process is explained here in the https://blogs.technet.com/b/exchange/archive/2012/08/14/calendar-and-tasks-retention-tag-support-in-exchange-2010-sp2-ru4.aspx.
If you do not want calendar and tasks folder to process follow the Work around in the blog post before or after deploying RU4 or later SP, RU ( If you
are upgrading from sp2 or RU3), items in calendar and tasks folder are stamped with Default Policy Tag (DPT).

Soon you notice that calendar and tasks items are stamped with retention tag, you might try follow the work around from the blog post by adding Registry key ( Or disabled RPTs).

Adding registry key or disabled RPTs prevent further stamping of calendar and tasks items. But the items already stamped still contain stamped DPT expiry information Which is displayed in Outlook. And items already stamped with DPT show as "expired" passing the retention time but will not be removed from the calendar or tasks because Mailbox assistants process stops processing tasks and calendar items after adding registry key.

Now it is cosmetic issue, where items are stamped with retention tag but not doing anything. For many exchange users this is confusing.

Q. How to fix the cosmetic issue, ( remove the Default policy tag (DPT) information form the items prevent further stamping items .

To remove the retention tag, items must be reprocessed by the assistants process. To process the items you need to enable registry key which was disabled earlier. Before enable the registry key again follow the workaround below

Workaround:

- Apply retention hold on mailboxes : https://technet.microsoft.com/en-us/library/dd335168. - Create a retention policy , do not add any retention tags into the policy. Its like an empty policy

- Apply Retention policy with no tags included , to affected user mailboxes. Allow MFA to process process mailboxes.

- Remove the mailboxes from retention hold

- If you added registry key to disable processing of Calendar/Tasks items, enable it.

* you must allow sufficient time for AD replication , while adding and removing mailboxes to retention hold.

- Disable registry key (or add disabled RPTs for Calendar & Tasks), reapply the old policy

Caveats

1. Workaround explained above fix the cosmetic issue, but items are stamped again and processing will be delayed

2. you might see increase in database size and storage used , because items are stamped a fresh

Q. What if you do not want to follow the workaround .

- Items already stamped still contain stamped DPT expiry information Which is displayed in Outlook as "expired" after passing the retention time

 - Its only a cosmetic where items display "expired" information in outlook but it will not be removed from calendar.

-Manju