Graph API InternalServerError with FanoutDownstreamContradiction when running queries with large amount of results over multiple sites

Question

Hi,

I am running a search query against our SharePoint environment over a large amount of sites using the Graph API. For this we use the 'search/query' endpoint. The request is done using delegated permissions. Currently, when we run a query, most of the times it goes well, but sometimes gives an error with a statuscode 500 like this:

{'error': {'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'FanoutDownstreamContradiction', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'TwoStepFanout_FirstStepFailed', 'message': 'The call failed, please try again.', 'target': '', 'serviceName': 'Xap', 'moduleName': 'SubstrateSearch.FanoutV2.Plugins.TenantSearch.TenantSearchMultiDimensionFanoutPluginV3', 'contactTeam': '3sdri', 'httpCode': 500}, {'code': 'FanoutDownstreamContradiction', 'message': 'The call failed, please try again.', 'target': '', 'serviceName': 'FanoutService', 'moduleName': 'Fanout', 'contactTeam': '3STenantSearchDevs', 'httpCode': 500}], 'serviceName': 'FanoutService', 'moduleName': 'Fanout', 'contactTeam': '3STenantSearchDevs', 'httpCode': 500}], 'moduleName': 'SubstrateMultiDimensionFanoutSearchWorkflowBaseV2', 'httpCode': 500}], 'moduleName': 'G21AscWorkflow', 'httpCode': 500}], 'moduleName': 'TenantFileSearchFederationWorkflow_ASC', 'httpCode': 500}], 'moduleName': 'TenantFileSearchFederationWorkflow', 'httpCode': 500}], 'warnings': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'ProviderSearchRequestNull', 'message': 'The call failed, please try again.', 'target': '', 'serviceName': 'Xap', 'moduleName': 'TopicEngineQueryRequestBuilderPlugin', 'contactTeam': '3sdri', 'httpCode': 500}], 'moduleName': 'HttpWorkflow', 'httpCode': 500}, {'code': 'InternalServerError', 'message': 'The call failed, please try again.', 'target': '', 'details': [{'code': 'EmptyResponses', 'message': 'The call failed, please try again.', 'target': '', 'serviceName': 'TopicEngine', 'moduleName': 'TopicEngineQueryResponseValidatorPlugin', 'contactTeam': '3sdri', 'httpCode': 500}], 'moduleName': 'HttpWorkflow', 'httpCode': 500}], 'moduleName': 'HttpWorkflow', 'httpCode': 500}], 'moduleName': 'TopicEngineWorkflowWithTriggering', 'httpCode': 500}], 'moduleName': 'FederationWorkflow', 'httpCode': 500}], 'moduleName': 'TopLevelWorkflowBase', 'httpCode': 500}, 'Instrumentation': {'TraceId': '36560a36-0a8a-4fda-7619-320942eab818'}}

This problem has only started to occur recently, whereas before we have never had an issue with the queries. It seems to be most common when a lot of documents are being found across many sites for a a search query. The search query itself looks like this:

(FileName:*annual report* OR Title:*annual report*) filetype:pdf LastModifiedTimeForRetention=Yesterday

And thus errors depending on the amount of documents that are being found. It is totally unclear from the error what the problem is, and I'm wondering if anyone has seen this problem and if there is a possible solution for it...