Why does the chat completion API response stop early when using a custom content filter/data?

Hello Duncan Eley,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you are having issue with custom filter in Asynchronous mode.

Solution

The issue you're encountering with the custom content filter in streaming mode using Asynchronous Filter seems to be related to how the filter interacts with the streaming process and the integration with Azure AI Search.

You can check the following to resolve the issue:

• Review your custom content filter's logic is compatible with streaming mode. Asynchronous filtering might be causing a delay or interruption in the stream, leading to the premature stop.
• Incrementally test each part of your request and filter logic to identify the exact cause. This includes trying different configurations of the data source and filter settings.
• Also, ensure all components (model, content filter, Azure AI Search) are up-to-date with the latest patches and updates from Azure.
• Lastly, use the example below to adjust your request to see if simplifying or changing certain parameters helps.

{
    "messages": [
        {
            "role": "user",
            "content": "initial question"
        },
        {
            "role": "assistant",
            "content": "initial response"
        },
        {
            "role": "user",
            "content": "follow up"
        }
    ],
    "temperature": 0.0,
    "max_tokens": 500,  // Reduce max tokens for testing
    "top_p": 1.0,
    "stop": null,
    "stream": true,
    "model": "my-deployment",
    "extra_body": {
        "data_sources": [
            {
                "type": "azure_search",
                "parameters": {
                    "endpoint": "https://my-search-endpoint.search.windows.net",
                    "authentication": {
                        "type": "api_key",
                        "api_key": "*****"
                    },
                    "index_name": "my-search-index",
                    "fields_mapping": {
                        "content_fields": [],
                        "title_field": null,
                        "url_field": null,
                        "filepath_field": null,
                        "vector_fields": []
                    },
                    "in_scope": false,
                    "top_n_documents": 5,
                    "query_type": "simple",
                    "semantic_configuration": "",
                    "role_information": "my role info",
                    "filter": null,
                    "strictness": 3
                }
            }
        ]
    }
}

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam