Azure OpenAI returns only short completions

Manu Cohen-Yashar 71 Reputation points
2022-08-31T14:22:14.52+00:00

I am evaluating the Azure OpenAI service for a use case of summarization and QnA.
I noticed that the content_filter blocks the response from the service when I configure the max_tokens to be larger than 100. In some cases, even a value over 10 blocks the response.
In our use case, we need to summarize a large number of trouble tickets into a one-pager executive summary. The text we need to generate is not "long" but is much longer than 100 words.
Are there any guidelines or procedures that we can follow to tune the content_filter?

Here is an example of the response we get:

{  
        "id": "cmpl-5kzRCnvZ2cgEpOGENxrgJtfVOH20c",  
        "object": "text_completion",  
        "created": 1661880938,  
        "model": "text-davinci-002",  
        "choices": [  
            {  
                "text": "",  
                "index": 0,  
                "finish_reason": "content_filter",  
                "logprobs": null  
            }  
        ]  
    }  

Thanks

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,124 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Chris Hoder - MSFT 96 Reputation points Microsoft Employee
    2022-09-28T01:08:39.237+00:00

    Hi,

    The system does not explicitly filter on any generation of larger than 100 tokens. It will evaluate the generated text to see if any flagged content was generated. You can read more about the behavior of the system here: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/content-filter.

    In general, I'd recommend a looking at the reason for the filtering to determine the next steps:

    • For prompts being filtered the service will return a 400 error, you could consider some flow to suggest the user change the input
    • When generated content is filtered (Such as in the example above where the finish_reason is content filter), I'd recommend adding in retry logic as this may be only for a specific generation. You will also want to consider a maximum number of retries

    We did ship a change to the content filters earlier this month to reduce false-positives in the filtering. Are you still observing this behavior?

    0 comments No comments