Azure OpenAI returns only short completions

Question

I am evaluating the Azure OpenAI service for a use case of summarization and QnA.
I noticed that the content_filter blocks the response from the service when I configure the max_tokens to be larger than 100. In some cases, even a value over 10 blocks the response.
In our use case, we need to summarize a large number of trouble tickets into a one-pager executive summary. The text we need to generate is not "long" but is much longer than 100 words.
Are there any guidelines or procedures that we can follow to tune the content_filter?

Here is an example of the response we get:

{  
        "id": "cmpl-5kzRCnvZ2cgEpOGENxrgJtfVOH20c",  
        "object": "text_completion",  
        "created": 1661880938,  
        "model": "text-davinci-002",  
        "choices": [  
            {  
                "text": "",  
                "index": 0,  
                "finish_reason": "content_filter",  
                "logprobs": null  
            }  
        ]  
    }  

Thanks

Answer 1

Hi,

The system does not explicitly filter on any generation of larger than 100 tokens. It will evaluate the generated text to see if any flagged content was generated. You can read more about the behavior of the system here: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/content-filter.

In general, I'd recommend a looking at the reason for the filtering to determine the next steps:

• For prompts being filtered the service will return a 400 error, you could consider some flow to suggest the user change the input
• When generated content is filtered (Such as in the example above where the finish_reason is content filter), I'd recommend adding in retry logic as this may be only for a specific generation. You will also want to consider a maximum number of retries

We did ship a change to the content filters earlier this month to reduce false-positives in the filtering. Are you still observing this behavior?