Chat in Azure AI Studio fails when max tokens set at 5000

matsuo_basho 30

I'm using the Chat playground in Azure AI Studio, running Meta Llama 3.1 70B instruct with a Serverless compute.

When I set the max tokens to 500, it works fine. However, when I set it to 5000, I get an error:
Request failed with status code 500. Clear the output to start a new dialog.

According to Meta's announcement, the model's context length is 128K, so why is there an issue?

User's image

Please let me know if this is some sort of MS quota issue and if so, whether it's something I have to request and wait for. Perhaps it's the fact that I'm running this on serverless compute.

YutongTie-MSFT 52,776 Reputation points

2024-07-26T17:13:27.3133333+00:00

Thanks for reaching out to us, just want to confirm it's 5000 as the token right? We are checking internally for this issue.

Regards,

Yutong
matsuo_basho 30 Reputation points

2024-07-26T18:39:06.13+00:00

Correct. 2500 still seems to work, but at 5000 I get the error.

Note that I actually need it to go up to hundreds of thousands (or whatever the maximum context length of the given model is).
matsuo_basho 30 Reputation points

2024-07-29T15:53:05.03+00:00

Any update on this? Hope to get an update within 1 day or I'll have to abandon using Azure with this and spend time on finding another solution, which I would rather not do.
YutongTie-MSFT 52,776 Reputation points

2024-07-29T15:56:48.0833333+00:00

Hello @matsuo_basho

Thanks for your response, I would like to suggest you raise a support ticket to check on the backend data. Please let us know if you have no support plan, I am happy to enable you a free ticket for this case.

Please let us know.

Regards,

Yutong
matsuo_basho 30 Reputation points

2024-07-29T16:05:38.27+00:00

I do not have a support plan, please enable a ticket, thanks.
YutongTie-MSFT 52,776 Reputation points

2024-07-29T16:07:29.2833333+00:00

Please share your Azure Subscription ID in the private message.

Accepted answer

YutongTie-MSFT 52,776 Reputation points

2024-07-29T16:51:54.5733333+00:00

Hello Mastuo,

Thanks for reaching out to us again, since this issue need to be checked with more details, we have redirect it to support team. Please kindly accept the answer here so that we can close this issue in the public forum.

Once you get your issue solved, please post the solution here if it is convenient to share.

I hope it helps.

Regards,

Yutong
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

matsuo_basho 30 Reputation points

2024-08-01T10:44:42.4766667+00:00

As of now, this is a bug with no solution since the Microsoft team has been investigating this for a few days.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Chat in Azure AI Studio fails when max tokens set at 5000

1 additional answer

Your answer