How do I get the logprobs with the Chat Completion Model gpt-4-turbo-2024-04-09 with 2024-06-01 api version?

Question

How do I get the logprobs with the Chat Completion Model gpt-4-turbo-2024-04-09 with 2024-06-01 api version?

Yash Runwal 0

I have deployed a gpt-4-turbo-2024-04-09 model in Azure OpenAI Service and am constructing the payload with the logprobs parameter as True. But in the response, the logprobs are absent. But according to the following documentation, it should be available with the 2024-06-01 api version.

https://learn.microsoft.com/en-us/azure/ai-services/openai/whats-new

So what am I missing here?

Yash Runwal 0

I have a payload that also contains image data. Without it, I get the logprobs as well. Sorry for not mentioning it beforehand

My payload structure also includes images. Following is a snippet where payload is created.

    def _get_payload(self, system_prompt, user_prompt): 
        payload = { 
            "model": "gpt-4-turbo-2024-04-09", 
            "logprobs": True, 
            "temperature": 0, 
            "messages": [ 
                { 
                    "role": "system", 
                    "content": [ 
                        { 
                            "type": "text", 
                            "text": system_prompt 
                        } 
                    ] 
                }, 
                { 
                    "role": "user", 
                    "content": [ 
                        { 
                            "type": "text", 
                            "text": user_prompt 
                        } 
                    ] 
                } 
            ], 
            "max_tokens": self.config.max_token 
        } 
        return payload 
 
    def _create_payload(self, system_prompt, user_prompt, encoded_pages): 
        payload = self._get_payload(system_prompt, user_prompt) 
        for message in payload['messages']: 
            if message['role'] == 'user': 
                for encoded_page in encoded_pages: 
                    image_entry = { 
                        "type": "image_url", 
                        "image_url": { 
                            "url": f"data:image/png;base64,{encoded_page}" 
                        } 
                    } 
                    message['content'].append(image_entry) 
                break 
        return payload

@Pavankumar Purilla @Sina Salam

Sina Salam 22,031 Reputation points Volunteer Moderator

2024-11-19T21:41:19.11+00:00
Hello Yash Runwal,

Thank you for your feedback.

I can still affirm that, As of the API version 2024-06-01, logprobs is not supported in the Chat Completion API. It is a feature exclusive to the Completions API, which is not compatible with models like gpt-4-turbo.

The best alternative pproaches if logprobs is essential:

Switch to the Completions API: Use a model such as text-davinci-003 or similar that supports logprobs.

response = openai.Completion.create( model="text-davinci-003", prompt="Your prompt here", max_tokens=100, logprobs=5 ) print(response)

If Chat Completion features are required: You'll need to rely on probabilities implicitly (like using temperature, top_p, or nucleus sampling) for fine-tuned control, but logprobs output remains unavailable.

Now, integrating image data further complicates the use case since image embedding or multimodal inputs may require using separate APIs or methods specific to the Azure OpenAI model's capabilities. I will recommend, if you require both logprobs and chat-based interactions, it's worth contacting Azure OpenAI support by raising ticket to clarify feature availability or exploring if future API versions support this hybrid need.

Success.
Pavankumar Purilla 8,570 Reputation points Microsoft External Staff Moderator

2024-11-22T22:05:51.1366667+00:00

Hi Yash Runwal,
Sorry for the late reply. Just checking in to see if the above response provided by @Sina Salam helped.

2 answers

Your answer

Yash Runwal 0 Reputation points

2024-11-19T21:13:04.63+00:00

I have a payload that also contains image data. Without it, I get the logprobs as well. Sorry for not mentioning it beforehand

My payload structure also includes images. Following is a snippet where payload is created.

def _get_payload(self, system_prompt, user_prompt): payload = { "model": "gpt-4-turbo-2024-04-09", "logprobs": True, "temperature": 0, "messages": [ { "role": "system", "content": [ { "type": "text", "text": system_prompt } ] }, { "role": "user", "content": [ { "type": "text", "text": user_prompt } ] } ], "max_tokens": self.config.max_token } return payload def _create_payload(self, system_prompt, user_prompt, encoded_pages): payload = self._get_payload(system_prompt, user_prompt) for message in payload['messages']: if message['role'] == 'user': for encoded_page in encoded_pages: image_entry = { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{encoded_page}" } } message['content'].append(image_entry) break return payload

@Pavankumar Purilla @Sina Salam
Sina Salam 22,031 Reputation points Volunteer Moderator

2024-11-19T21:41:19.11+00:00

Hello Yash Runwal,

Thank you for your feedback.

I can still affirm that, As of the API version 2024-06-01, logprobs is not supported in the Chat Completion API. It is a feature exclusive to the Completions API, which is not compatible with models like gpt-4-turbo.

The best alternative pproaches if logprobs is essential:

Switch to the Completions API: Use a model such as text-davinci-003 or similar that supports logprobs.

response = openai.Completion.create( model="text-davinci-003", prompt="Your prompt here", max_tokens=100, logprobs=5 ) print(response)

If Chat Completion features are required: You'll need to rely on probabilities implicitly (like using temperature, top_p, or nucleus sampling) for fine-tuned control, but logprobs output remains unavailable.

Now, integrating image data further complicates the use case since image embedding or multimodal inputs may require using separate APIs or methods specific to the Azure OpenAI model's capabilities. I will recommend, if you require both logprobs and chat-based interactions, it's worth contacting Azure OpenAI support by raising ticket to clarify feature availability or exploring if future API versions support this hybrid need.

Success.
Pavankumar Purilla 8,570 Reputation points Microsoft External Staff Moderator

2024-11-22T22:05:51.1366667+00:00

Hi Yash Runwal,
Sorry for the late reply. Just checking in to see if the above response provided by @Sina Salam helped.

Answer 1

Hello Yash Runwal ,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to know how you can get the logprobs with the Chat Completion Model gpt-4-turbo-2024-04-09 with 2024-06-01 api version.

No, you cannot get logprobs with the Chat Completion Model gpt-4-turbo-2024-04-09 even with the 2024-06-01 API version. The logprobs parameter is supported for the completions API but not for the chat completions API. This is likely why you are not seeing logprobs in your response.

If you need logprobs, you should use the completions API instead of the chat completions API. This is an example of how to construct your payload for the completions API:

import openai
response = openai.Completion.create(
    model="gpt-4-turbo-2024-04-09",
    prompt="Your prompt here",
    max_tokens=100,
    logprobs=5,
    api_version="2024-06-01"
)
print(response)

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Answer 2

Hi Yash Runwal,

Greetings & Welcome to the Microsoft Q&A forum! Thank you for posting your query.

I understand that you are facing an issue with logprobs not appearing in the response from the Azure OpenAI service using the GPT-4-Turbo model (gpt-4-turbo-2024-04-09) with the 2024-06-01 API version.

I have tested this scenario using the same model and API version (2024-06-01), and I was able to successfully retrieve logprobs in the response.
Here's a working example of a correct API request:

curl -X POST https://<resourcename>.openai.azure.com/openai/deployments/deploymentname/chat/completions?api-version=2024-06-01 ^
-H "Content-Type: application/json" ^
-H "api-key: <API KEY>" ^
-d "{\"messages\":[{\"role\":\"user\",\"content\":\"What is the capital of France?\"}],\"max_tokens\":10,\"temperature\":0,\"logprobs\":true,\"top_logprobs\":5}"

Here is the screenshot of output:
User's image Here are the some troubleshoot steps:

Ensure the deployed model is gpt-4-turbo-2024-04-09.
Confirm the API version is set to 2024-06-01.
Use "logprobs": true in the request payload.
Double-check the deployment name in the API URL matches your deployment.
Ensure your subscription has sufficient quota and necessary permissions.
Test the same payload with another deployment or region to see if results vary.
If still unresolved, contact Azure support with resource details, payload, response, and error logs.

Please refer the following documentation: createChatCompletionRequest.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful.

Share via

How do I get the logprobs with the Chat Completion Model gpt-4-turbo-2024-04-09 with 2024-06-01 api version?

2 answers

Your answer