All queries from from FHIR server to Cosmos DB failing

Question

We are running an instance of the Azure FHIR Server using Cosmos DB. Since at least 12 hours ago, all queries to the FHIR server are failing with the following exception and trace:

2024-05-08T05:41:26.650255158Z fail: Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore[0]
2024-05-08T05:41:26.650270158Z CosmosFhirDataStore. 'Search Parameters' failed. Elapsed: 00:00:00.0006074. Memory: 186966472.
2024-05-08T05:41:26.650273858Z Microsoft.Health.Fhir.Core.Exceptions.InitializationException: Failed to initialize search parameters
2024-05-08T05:41:26.650277358Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.EnsureInitialized() in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 106
2024-05-08T05:41:26.650280958Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 112
2024-05-08T05:41:26.650284658Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchableSearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchableSearchParameterDefinitionManager.cs:line 49
2024-05-08T05:41:26.650310959Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParamsAsync(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 185
2024-05-08T05:41:26.650314759Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParametersAsync() in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 348
2024-05-08T05:41:26.650318059Z at Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore.Build(ICapabilityStatementBuilder builder) in /repo/src/Microsoft.Health.Fhir.CosmosDb/Features/Storage/CosmosFhirDataStore.cs:line 806
2024-05-08T05:41:26.650655764Z fail: Microsoft.Health.Fhir.Core.Features.Conformance.SystemConformanceProvider[0]
2024-05-08T05:41:26.650665064Z SystemConformanceProvider: Failed running 'Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore' when building a new CapabilityStatement.
2024-05-08T05:41:26.650668964Z Microsoft.Health.Fhir.Core.Exceptions.InitializationException: Failed to initialize search parameters
2024-05-08T05:41:26.650672264Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.EnsureInitialized() in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 106
2024-05-08T05:41:26.650675665Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs:line 112
2024-05-08T05:41:26.650678965Z at Microsoft.Health.Fhir.Core.Features.Definition.SearchableSearchParameterDefinitionManager.GetSearchParameters(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Definition/SearchableSearchParameterDefinitionManager.cs:line 49
2024-05-08T05:41:26.650682665Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParamsAsync(String resourceType) in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 185
2024-05-08T05:41:26.650686065Z at Microsoft.Health.Fhir.Core.Features.Conformance.CapabilityStatementBuilder.SyncSearchParametersAsync() in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/CapabilityStatementBuilder.cs:line 348
2024-05-08T05:41:26.650689565Z at Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosFhirDataStore.Build(ICapabilityStatementBuilder builder) in /repo/src/Microsoft.Health.Fhir.CosmosDb/Features/Storage/CosmosFhirDataStore.cs:line 806
2024-05-08T05:41:26.650692965Z at Microsoft.Health.Fhir.Core.Features.Conformance.SystemConformanceProvider.GetCapabilityStatementOnStartup(CancellationToken cancellationToken) in /repo/src/Microsoft.Health.Fhir.Core/Features/Conformance/SystemConformanceProvider.cs:line 146

The response content is the following:

{
    "resourceType": "OperationOutcome",
    "id": "ee3a3476-4e9c-4f28-a960-699f23b1cab6",
    "meta": {
        "lastUpdated": "2024-05-08T05:51:35.2237241+00:00"
    },
    "issue": [
        {
            "severity": "error",
            "code": "unknown",
            "diagnostics": ""
        }
    ]
}

We are not sure if the issue lies with the FHIR server instance or the connection with the Cosmos DB instance.

We have 3 environments with the same setup and all of them have the same issue. We have not deployed any updates or made any changes that could explain these issues.