7,925 questions
To anybody else looking at this post, we didn't have to do an OS reinstall, but we did have to reinstall Exchange and everything is back to normal now, even after all that troubleshooting listed above.
Database copy content indices keep failing, service not started, but it is
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 68%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJL%3C/text%3E%3C/svg%3E)
Joseph Larrew
341
Reputation points Microsoft Employee
I’ve run into an issue where a customer’s content indices for database copies on a particular server go back and forth between “Unknown” and “Failed.” The error on the database copy says the Host Controller service isn’t started, but it clearly is. Restarting the service doesn’t make anything happen. Updating the copy with the -CatalogOnly switch doesn’t do anything. I’ve also completely removed the copy, files and all, then re-added the copy and the content indices are still not in a healthy state. Looking at the event log while the content index is in a failed state, I see a couple of seemingly relevant errors:
1) Event 1010 – MSExchangeFastSearch - “Could not connect to net.tcp://localhost:3803…TCP Error Code 10061 - No connection could be made because target machine actively refused it…”
2) Event 1009 – MSExchangeFastSearch – “Indexing of [mailbox database] encountered an unexpected exception…The component operation failed…error code: ‘FastError’…”
I checked IIS bindings for both front end and back end and the correct certs are there and assigned as well. Checking port “3803” to the source servers is also fine. The server with the issue is actually in its own subnet and the DAG network shows that subnet as “Up” as well. This has been happening since 31 AUG. I think there was some sort of network blip that caused this particular server to take control of the PAM, activate its database copies, etc. DAC mode isn't on yet, so we're looking at that as well. It should be noted that when the content index status is failed, I see the errors and events above. When it is in an "Unknown" status, there is no error indicated from the Get-MailboxDatabaseCopyStatus cmdlet, but in the event log, there are a couple of relevant events:
1) Event ID 1026 - .NET Runtime - Application: NodeRunner.exe, Framework Version: v4.0.30319, Description: The process was terminated due to an unhandled exception. ExceptionInfo: SystemAccessViolation at Microsoft.Ceres.SearchCore.FastServer.Plugin.CreateIndexer()...."
2) Event ID 1000 - Application Error - Faulting Application Name: Noderunner.exe; version: 16.0.1497.0; timestamp: 0x5cb8eb2d; ExceptionCode: 0xc0000005....FaultingModulePath: %EXCHANGEINSTALLPATH%\Bin\Search\Ceres\Native\Microsoft.Ceres.SearcCore.FastServer.Native.dll; Report Id: <long hex string>..."
Exchange | Exchange Server | Management
{count} votes
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-09T11:38:07.273+00:00 Also, I see a couple more Event ID 1000's with different Faulting Module names. In total, there is ntdll.dll; Microsoft.Ceres.SearchCore.FastServer.Native.dll; KERNELBASE.dll; MSVCR110.dll. Each one of these event ID include a "Report Id" with a long hex string as an identifier. Any idea where these "reports" might be?
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-09T11:58:48.97+00:00 Another interesting development is that I've found that while this server is a part of a DAG, it uses a different namespace as part of a "continuity of operations" plan. The namespace is not on the certificate. Also, all the servers within the DAG have the ASA credentials rolled out, except for this one because, according to the famous "configure kerberos for load balanced client access..." article, it needs to be configured for load balanced namespaces, and this one is not load balanced... I say all this, but this issue has only been occurring since a network blip and the ASA configuration has been like this for some time.
Sign in to comment
Accepted answer
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-21T17:12:34.603+00:00
2 additional answers
Sort by: Most helpful
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 71%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELZ%3C/text%3E%3C/svg%3E)
Lydia Zhou - MSFT 2,386 Reputation points Microsoft Employee2020-09-10T02:38:30.193+00:00 anonymous user
What's the detailed version of your DAG members? How many DAG members do you have? Please provide more details about your environment.
Do you install any software or update on the server before this issue?Please use the following command to make sure the search index is enabled, and check the content index for all database copies:
Get-MailboxDatabase | Format-Table Name,IndexEnabled,servername Get-MailboxDatabaseCopyStatus *| select name,status,contentindexstate|Sort-Object nameExcept Microsoft Exchange Search Host Controller service, please also check and insure that Microsoft Exchange Search service is running well.
When this database copy is mounted on other DAG member, does the search index issue still occur?
Please also try to mount other database copies on this server, check if the search index issue can be reproduced on other databases. We have to make sure if this issue is related to a specific DAG member server, or a particular database copy.You can try to remove the database copy on this server, and re-add it. Then mount the new db copy to check again.
Additionally, you can post the screenshot of the error event 1010 and 1009 for further analysis, and don't forget to cover your personal information.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-10T16:54:12.38+00:00 Hi Lydia, I'm going to send this link to my coworkers so that they can get this info for you, as I'm away until next week. Here's what I can tell you without running any commands:
DAG has 6 members and an FSW. They are all on Exchange 2013 CU23, .NET version is 4.8, Server 2012R2. Nothing has been installed recently, except for maybe some patches. The A/V has been disabled as a test as well, still nothing. Activating copies on the server doesn't fix the CI, it still says whatever all the other copies on that server say. None of the other copies on other servers within the DAG have this issue, but all the copies on a specific server do. Yes, both services are running/have been restarted. We have removed a database copy on that server a couple of times to rebuild them, only to still have the CI be failed or unknown state still.
-
Lydia Zhou - MSFT 2,386 Reputation points Microsoft Employee
2020-09-11T06:59:12.663+00:00 anonymous user
That's OK. We look forward to the reply from your coworkers. Hope we can get more information from the event logs.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread. -
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-14T15:05:59.65+00:00 See attachments for the additional info requested.[24581-eventid1010.txt][1] [1]: /answers/storage/temp/24581-eventid1010.txt
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-14T16:36:32.13+00:00 Event ID 1009:
The indexing of mailbox database <site name>-ADAG-DB11 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Engine.FeedingSkippedException: "Feeding was skipped for '5809f4b1-ff83-4ce8-b1d2-2e42e52ff7a4 (<site name>-ADAG-DB11)' due to the state 'Unknown', error code: 'Unknown'."
at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()
at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult) -
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-14T16:51:01.567+00:00 See the attached time frame for event ID 1010 ![24524-eventid1010.png][1] [1]: /answers/storage/temp/24524-eventid1010.png
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-14T16:56:14.243+00:00 I'm having pretty big issues submitting replies with attachments, which is why there are so many, just FYI. I'm just gonna say that the time frame for Event ID 1009 is every one per database copy on that server every 5 minutes or so.
Sign in to comment -
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-15T12:23:09.143+00:00 This is what comes up when I try to do an "Update-MailboxDatabaseCopy -CatalogOnly":
WARNING: Seeding of content index catalog for database <database name> failed. Please verify that the Microsoft Search
(Exchange) and the Host Controller service for Exchange services are running and try the operation again. Error: There
was no endpoint listening at net.tcp://localhost:3863/Management/SeedingAgent-0870A2FD-AD68-4B14-9CD6-AAB4FF4793FF12/Single that could accept the
message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details..Those services are indeed running on all servers. I ran a test-netconnection to each of my servers from all of my servers over ports 3800-3803,3863 and a couple of others and they all worked. I checked a netstat -a and found that the server was actually listening on those ports as well (both with 0.0.0.0:[port] and 127.0.0.1:[port]) so it doesn't look like it's that. I don't know what it means by "incorrect address or SOAP" and I don't know where to find an "InnerException." This sounds like IIS though.
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-15T17:44:48.393+00:00 So I've also tried rebuilding the Search Foundation and removing "Content Submitters" as an authorized role per another post on social TechNet and rebuilding the CI from there, still with no luck. Error I receive is:
WARNING: Seeding of content index catalog for database [database name] failed. Please verify that the Microsoft Search (Exchange) and the Host Controller service for Exchange services are running and try the operation again. Error: Could not connect to net.tcp://localhost:3863/Management/SeedingAgent-1B819314-D028-4EBF-9C26-C7936B94FE3B12/Single. The connection attempt lasted for a time span of 00:00:02.0157729. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:3863.
Q: I've seen this error more than once, so why is the target machine actively refusing it? (if that's actually why it is failing)
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-15T18:39:07.707+00:00 I've tried disabling Windows Firewall and HIPS, but no love still.
-
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-15T18:43:31.777+00:00 I've also tried, disabling IPv6 and preferring IPv4 over IPv6 via the registry and still no luck. I either get "target machine refused" or "incorrect address or SOAP action" mentioned above.
-
Lydia Zhou - MSFT 2,386 Reputation points Microsoft Employee
2020-09-16T09:35:55.92+00:00 anonymous user
Do you try to remove this Exchange server from DAG, then add it back?
If the issue persists after re-adding the DAG member, please try to recreate the Exchange content index catalog.
You can follow these steps: Reseed the content index catalog if there is only one copy of the mailbox database.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread. -
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-16T16:32:07.097+00:00 I guess removing it from the DAG is the next option, then reinstalling Exchange at this point. I'm kinda tired of messing with for something as small as a content index.
-
Lydia Zhou - MSFT 2,386 Reputation points Microsoft Employee
2020-09-17T05:20:54.647+00:00 anonymous user
Does renaming to recreate the folder that contains the Exchange content index catalog work?
If it doesn't work as well, you may try to re-configure Search Foundation with installconfig.ps1, and modify WcfConfigurator.xml on the server. You can check the detailed steps from these thread: Exchange 2013 Content Index status "unknown".
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread. -
Lydia Zhou - MSFT 2,386 Reputation points Microsoft Employee
2020-09-17T05:21:46.797+00:00 Since something may be corrupted on the server, yes, as the last suggestion, we have to remove this Exchange, and add a new DAG member.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread. -
Joseph Larrew 341 Reputation points Microsoft Employee
2020-09-17T12:44:26.927+00:00 Yes, I had tried rebuilding the Search Foundation like you mentioned, and still no luck. That's why we are rebuilding the server. The network connection isn't so great, so it's gonna take a while to replicate all the database copies down there....
-
Lydia Zhou - MSFT 2,386 Reputation points Microsoft Employee
2020-09-18T08:35:50.107+00:00 Hope your new server and database copies can work as expected, and we look forward to your good news.
If you have other issues about Exchange, please feel free to post with Exchange tags.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
Sign in to comment -