SharePoint 2019 critical event ID 6398 every 10 minutes.

Question

I have SP2019 farm (v. 16.0.10366.12106) where all roles exept SQL are on the same server. I get two errors every 10 or 20 minutes minutes.
Every 10 minutes:

1. The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition threw an exception. More information is included below.
   Failed to run flow Microsoft.CustomDictionaryDeployment.

Every 20 minutes:
2. The Execute method of job definition Microsoft.Office.Server.Search.Administration.QueryClassificationDictionaryUpdateTimerJobDefinition threw an exception. More information is included below.
Failed to run flow Microsoft.QueryClassificationDictionaryCompilationFlow.

I see also two timer jobs which fail at the same time:

1. Query Classification Dictionary Update for Search Application Search Service Application. (Failed to run flow Microsoft.QueryClassificationDictionaryCompilationFlow)
2. Search Custom Dictionaries Update (Failed to run flow Microsoft.CustomDictionaryDeployment.)

ULS logs:
Custom dictionary deployment timer job failed. SearchComponent: OWSTIMER.EXE, SearchServiceApplication: {94869140-9f09-4397-a7ac-20804119d25f}, FlowName: Microsoft.CustomDictionaryDeployment, CorrelationId: {bda8819f-238a-e0ae-6058-c4efc9cd8ffb}, Failure: System.InvalidOperationException: Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: 72b8ec58-ad74-4268-a742-437e043a59bc. at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.DoExecute(CtsConnection connection) at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob(). bda8819f-238a-e0ae-6058-c4efc9cd8ffb
Custom dictionary deployment timer job failed. SearchComponent: OWSTIMER.EXE, SearchServiceApplication: {94869140-9f09-4397-a7ac-20804119d25f}, FlowName: Microsoft.CustomDictionaryDeployment, CorrelationId: {bda8819f-238a-e0ae-6058-c4efc9cd8ffb}, Failure: System.InvalidOperationException: Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: 72b8ec58-ad74-4268-a742-437e043a59bc. at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.DoExecute(CtsConnection connection) at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob(). bda8819f-238a-e0ae-6058-c4efc9cd8ffb
The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition (ID 16375a30-4ccb-4bda-af64-689726b4c340) threw an exception. More information is included below. Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: bda8819f-238a-e0ae-6058-c4efc9cd8ffb.. (Correlation=bda8819f-238a-e0ae-6058-c4efc9cd8ffb) bda8819f-238a-e0ae-6058-c4efc9cd8ffb
The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition (ID 16375a30-4ccb-4bda-af64-689726b4c340) threw an exception. More information is included below. Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: bda8819f-238a-e0ae-6058-c4efc9cd8ffb.. (Correlation=bda8819f-238a-e0ae-6058-c4efc9cd8ffb) bda8819f-238a-e0ae-6058-c4efc9cd8ffb
The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition (ID 16375a30-4ccb-4bda-af64-689726b4c340) threw an exception. More information is included below. Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: bda8819f-238a-e0ae-6058-c4efc9cd8ffb.. (Correlation=bda8819f-238a-e0ae-6058-c4efc9cd8ffb) bda8819f-238a-e0ae-6058-c4efc9cd8ffb
Exception stack trace: at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob() at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result) bda8819f-238a-e0ae-6058-c4efc9cd8ffb

ULS logs for ID 72b8ec58-ad74-4268-a742-437e043a59bc from the previou ULS log:
Dictionary deployment failed. SearchComponent: Content1-94869140-9f09-4397-a7ac-20804119d25f, CorrelationId: 72b8ec58-ad74-4268-a742-437e043a59bc, SearchServiceApplication: {00000000-0000-0000-0000-000000000000}, DictionaryName: Microsoft.UserDictionaries.EntityExtraction.Companies.Inclusions, Component:DictionaryRepositoryWriter, FailureMessage: error CS1606: Assembly signing failed; output may not be signed -- Error signing assembly -- The process cannot access the file because it is being used by another process. 72b8ec58-ad74-4268-a742-437e043a59bc
Dictionary deployment failed. SearchComponent: Content1-94869140-9f09-4397-a7ac-20804119d25f, CorrelationId: 72b8ec58-ad74-4268-a742-437e043a59bc, SearchServiceApplication: {00000000-0000-0000-0000-000000000000}, DictionaryName: Microsoft.UserDictionaries.EntityExtraction.Companies.Inclusions, Component:DictionaryRepositoryWriter, FailureMessage: error CS1606: Assembly signing failed; output may not be signed -- Error signing assembly -- The process cannot access the file because it is being used by another process. 72b8ec58-ad74-4268-a742-437e043a59bc
Dictionary deployment failed. SearchComponent: OWSTIMER.EXE, CorrelationId: 72b8ec58-ad74-4268-a742-437e043a59bc, SearchServiceApplication: {94869140-9f09-4397-a7ac-20804119d25f}, DictionaryName: Microsoft.UserDictionaries.EntityExtraction.Companies.Inclusions, Component:FlowExecution, FailureMessage: Deployment (flow:Microsoft.CustomDictionaryDeployment) for custom dictionary c866ca65-f095-4a16-9249-028d500f7703 did not complete successfully. Flow execution state was: Failed. Failure: Evaluation failure. An evaluation thread threw an exception. (Exception type: Microsoft.Ceres.Evaluation.DataModel.EvaluationException. Exception message: Evaluation failed in operator RepoWriter of type DictionaryRepositoryWriter), Cause: Microsoft.Ceres.ContentEngine.Services.ContentIntegrationEngine.FailCauseException: Evaluation failed in operator RepoWriter of t...
Custom dictionary deployment timer job failed. SearchComponent: OWSTIMER.EXE, SearchServiceApplication: {94869140-9f09-4397-a7ac-20804119d25f}, FlowName: Microsoft.CustomDictionaryDeployment, CorrelationId: {bda8819f-238a-e0ae-6058-c4efc9cd8ffb}, Failure: System.InvalidOperationException: Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: 72b8ec58-ad74-4268-a742-437e043a59bc. at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.DoExecute(CtsConnection connection) at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob(). bda8819f-238a-e0ae-6058-c4efc9cd8ffb

To fix this I:

1. Confirmed that the Managed Metadata Service Application, "This service application is the default storage location for keywords" and "This service application is the default storage location for column specific term sets" are checked.
2. Confirmed that search service account has full access to the Managed Metadata Service Application.
3. Reset the Index and do a full crawl.
4. Recreated Managed Metadata service.
5. Re-run the configuration wizard on the server.
6. Restarted server few times.

Unfortunately all these steps did not help.

Search service works fine, all components are healthy.
Managed metadata also works fine.

Answer 1

Hi @Oleg Tserkovnyuk ,

The error “The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition threw an exception” was caused by the failure of the Search Custom Dictionaries Update timer job.
The error “The Execute method of job definition Microsoft.Office.Server.Search.Administration.QueryClassificationDictionaryUpdateTimerJobDefinition threw an exception” was caused by the failure of the Query Classification Dictionary Update for Search Application Search Service Application timer job.

Firstly I suggest you to check whether SharePoint Search Host Controller Service is running on every server in your farm:

If they have been running, restart it. Then reset Index and do a full crawl, compare the result.

Those two timer jobs are all related to Search Dictionaries which you can find in Search Administration:

As you mentioned in your post, the search service works fine in your farm. If they still don’t work after you have tried the method above, I suggest you to disable those two timer jobs if they are not needed in your farm.

I hope this information has been useful, please let me know if you still need assistance.

---

If an Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.