Llama 3.1 405B Instruct as a serverless API not working with stream response

wong2 5

When calling it with the API, it works if stream is set to false: User's image

But if I set stream to true, the response is empty: User's image

dupammi 8,535 Reputation points Microsoft Vendor

2024-07-24T08:39:35.68+00:00

Hi @wong2

Thank you for your question.

It was stated as below in the documentation

Streaming allows the generated tokens to be sent as data-only server-sent events whenever they become available.

"Whenever they become available", suggests that while the feature is designed to stream data as it becomes available, there may be conditions or limitations.I request you to raise a support case through Azure portal. This will allow you to get assistance from Azure support in resolving the issue you're facing. Please provide a detailed description of the issue, including any error messages or screenshots, when raising the support case.
siewming.wong 5 Reputation points

2024-07-24T10:59:21.9066667+00:00

yes, not working, same as me, but when i logged every step, i found that this model return as usual after it supposed to stop but failed to stop i think, and return \n until exceeding 8000 token, might be maximum token. only when stream is true, false works fine
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Your answer