Does gpt-4-1106-preview support logprobs?

Question

I am using the openai Python package to import AzureOpenAI, and I have created a client for my gpt-4 deployment (1106-preview) with API version 2024-02-01. I am using Azure AI Search to perform RAG using the extra_body parameter, with a data source pointing to my vector database.

I would like to generate some confidence scores for all of the LLM responses, ideally using logprobs = True when using client.chat.completions.create. However, I keep receiving errors saying that log_probs is not supported, nor is logprobs, despite the documentation showing here: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions that it can be used.

Is this unsupported? Is there additional configuration for me to add prior to including logprobs=True in my client.chat.completions.create request?

Answer 1

Rohan Pathak I have checked internally and this parameter is not available in the new models, including gpt-4 and gpt-4-32k. Our team is actively working on supporting logprobs.

Also, there is no exact ETA which can be shared now.

Kindly keep an eye on the What's new in Azure OpenAI Service page for more updates.

Thanks for your understanding. Do let me know if you have any further queries.

Answer 2

Hello, I have the same issue as Rohan: logprobs are not available for Azure AI deployments on the chat completions endpoint.

What's the timeline on making logprobs available for gpt-4 and gpt-4o models ?

To prevent more headaches, could you also update the docs to say that this parameter is currently not supported ? Currently, this is really deceiving. https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#request-body-2

This is causing real issues for my client and blocking many use cases that require both logprobs and the Azure infrastructure. When will this feature be available?