KeyError: "usage" for gpt-3.5-turbo-16k

Question

KeyError: "usage" for gpt-3.5-turbo-16k

Hoang, Steven 40

I’ve recently been getting KeyErrors when using gpt-3.5-turbo-16k where the “usage” key is completely missing from the response object. What’s weird is that this happens inconsistently, without any changes to the code:

# OpenAI/Azure-specific authentication headers
headers = {
    "Authorization": await get_azure_token(),
    "OCP-Apim-Subscription-key": OPENAI_KEYS[model],
}

payload = {
	"messages": [message.model_dump() for message in messages],
    "temperature": kwargs.get("temperature", 1),
}
async with httpx.AsyncClient(
    verify=False, follow_redirects=True, timeout=360
) as client:
    resp = await client.post(url=OPENAI_URLS[model], json=payload, headers=headers)


Traceback (most recent call last):
...
    prompt_tokens=resp.json()["usage"]["prompt_tokens"],
                  ~~~~~~~~~~~^^^^^^^^^
KeyError: 'usage'

This is happening exclusively with 3.5. I am also using GPT4 models and they are working as expected.

navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-09T05:58:58.8166667+00:00
@Hoang, Steven Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

The KeyError: 'usage' you’re encountering with gpt-3.5-turbo-16k seems to be due to the usage key being absent from the response object. This can happen if the API is not returning usage data in the response.

Here are a few things you could consider:

Streaming Responses: If you’re streaming responses, the usage key might not be present.

Check the API Response: Log the entire response object or request content to check if the API is actually omitting usage. You can update your code like below:

prompt_tokens = resp.json()["usage"]["prompt_tokens"] if 'usage' in resp.json() else -1

This will assign -1 to prompt_tokens if usage is not in the response.

Please update the code accordingly and let me know if you encounter the same issue again. Hope this helps.
Sumit 30 Reputation points

2024-02-09T09:38:16.5133333+00:00
This key is found missed in response of other model's as well like gpt-35-turbo etc. It is intermittent in nature. We can handle this in code but the problem is we cant calculate cost incurred for each call if this key is missing. SO just assigning tokens to -1 is not the solution, maybe you can help us understanding why all of a sudden this key has started missing... Also other keys in the response are empty like

"created": 0, "id": "", "model": "", "object": ""
Hoang, Steven 40 Reputation points

2024-02-09T14:29:59.11+00:00

I agree with Sumit. It would be nice to be able to know why this behavior is not deterministic.I'm not passing any parameters to the chat completion call other than temperature (I modified my post to show the code leading up to the error).

Hoang, Steven 40

I agree with Sumit. We need the usage to track cost. Also, as of this morning, it is not happening intermittently anymore - now "usage" is always null. My JSON response is as follows:

print(resp.json())
{'choices': [{'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}, 'finish_reason': 'stop', 'index': 0, 'message': {'content': 'Two plus two equals four.', 'role': 'assistant'}}], 'created': 0, 'id': '', 'model': '', 'object': '', 'prompt_filter_results': [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}]}

I've also added code including the request. Would be nice to know why this is happening only on 3.5-turbo.

Sumit 30 Reputation points

2024-02-09T15:36:32.1733333+00:00

It is occurring for some of the deployments, for others it is giving usual response. For those deployments , those keys are being missed always as pointed out by Steven. Maybe some wrong branch is deployed in production in some of the regions ( a wild guess as it happens in tech companies frequently ) check canada-east region for example.
navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-12T04:45:13.9466667+00:00

@Sumit @Hoang, Steven Thanks for getting back. I am looking into this and I will get back to you with an update.
Sumit 30 Reputation points

2024-02-13T08:13:15.7033333+00:00

it has been fixed, thanks!
navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-13T08:16:44.6066667+00:00

@Sumit @Hoang, Steven Thanks for the confirmation. For the below answer, Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Accepted answer

0 additional answers

Your answer

navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-09T05:58:58.8166667+00:00

@Hoang, Steven Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

The KeyError: 'usage' you’re encountering with gpt-3.5-turbo-16k seems to be due to the usage key being absent from the response object. This can happen if the API is not returning usage data in the response.

Here are a few things you could consider:

Streaming Responses: If you’re streaming responses, the usage key might not be present.

Check the API Response: Log the entire response object or request content to check if the API is actually omitting usage. You can update your code like below:

prompt_tokens = resp.json()["usage"]["prompt_tokens"] if 'usage' in resp.json() else -1

This will assign -1 to prompt_tokens if usage is not in the response.

Please update the code accordingly and let me know if you encounter the same issue again. Hope this helps.
Sumit 30 Reputation points

2024-02-09T09:38:16.5133333+00:00

This key is found missed in response of other model's as well like gpt-35-turbo etc. It is intermittent in nature. We can handle this in code but the problem is we cant calculate cost incurred for each call if this key is missing. SO just assigning tokens to -1 is not the solution, maybe you can help us understanding why all of a sudden this key has started missing... Also other keys in the response are empty like

"created": 0, "id": "", "model": "", "object": ""
Hoang, Steven 40 Reputation points

2024-02-09T14:29:59.11+00:00

I agree with Sumit. It would be nice to be able to know why this behavior is not deterministic.I'm not passing any parameters to the chat completion call other than temperature (I modified my post to show the code leading up to the error).
Hoang, Steven 40 Reputation points

2024-02-09T14:55:57.9166667+00:00

I agree with Sumit. We need the usage to track cost. Also, as of this morning, it is not happening intermittently anymore - now "usage" is always null. My JSON response is as follows:

print(resp.json()) {'choices': [{'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}, 'finish_reason': 'stop', 'index': 0, 'message': {'content': 'Two plus two equals four.', 'role': 'assistant'}}], 'created': 0, 'id': '', 'model': '', 'object': '', 'prompt_filter_results': [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}]}

I've also added code including the request. Would be nice to know why this is happening only on 3.5-turbo.
Sumit 30 Reputation points

2024-02-09T15:36:32.1733333+00:00

It is occurring for some of the deployments, for others it is giving usual response. For those deployments , those keys are being missed always as pointed out by Steven. Maybe some wrong branch is deployed in production in some of the regions ( a wild guess as it happens in tech companies frequently ) check canada-east region for example.
navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-12T04:45:13.9466667+00:00

@Sumit @Hoang, Steven Thanks for getting back. I am looking into this and I will get back to you with an update.
Sumit 30 Reputation points

2024-02-13T08:13:15.7033333+00:00

it has been fixed, thanks!
navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-13T08:16:44.6066667+00:00

@Sumit @Hoang, Steven Thanks for the confirmation. For the below answer, Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 1

navba-MSFT 27,545 Microsoft Employee Moderator

@Sumit @Hoang, Steven I got an update from the Product Group team that the issue has been fixed now. Could you please test again and let me know if you are still encountering this issue ? Awaiting your reply.

** Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

navba-MSFT 27,545 Reputation points Microsoft Employee Moderator

2024-02-13T08:17:07.36+00:00

@Hoang, Steven For the above answer, Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
David Morton 0 Reputation points

2024-02-13T16:53:16.7433333+00:00

I have a gpt-4-vision-preview deployment in West US that is having the same issue, it is not fixed there?
Hoang, Steven 40 Reputation points

2024-02-16T20:05:46.1066667+00:00

Confirming that it works now. Thanks @navba-MSFT

Share via

KeyError: "usage" for gpt-3.5-turbo-16k

0 additional answers

Your answer