Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
1,281 questions
Usage details are not returned in the response when the stream is set to true in the chat/completions endpoint with AI RAG search.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 15%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Chandrashekar Machipeddi
0
Reputation points
Hi ,
We are using Azure AI Search with the chat/completions endpoint to search the data from the index. We are not getting the usage details, such as how many tokens are consumed. This issue occurs only when the stream is set to true.
Based on some blogs, we tried using the stream options like "stream_options": {"include_usage": true}.
However, this request failed with the error message: "Validation error at #/stream_options: Extra inputs are not permitted."
We are using gpt4 model with API version as 2024-10-21
POST https://xyxyxyx.openai.azure.com//openai/deployments/{DeployementName}/chat/completions?api-version=2024-10-21 HTTP/1.1
Is there any way to get the usage tokens with stream enabled.
Same thing is working fine, if we remove the data_sources parameter from the input body then we are getting the token usage., but to work with RAG search we have to pass the data_sources parameter too.
Below is the sample input body we are using for chat completion.
{"data_sources":[{"type":"azure_search","parameters":{"endpoint":"https://abcd.search.windows.net/","index_name":"developmentindex1","authentication":{"type":"api_key","key":"xxxxxxxxxxxxxxxxx"},"query_type":"simple","fields_mapping":{"content_fields_separator":"\n","content_fields":["TA_SYSTEM_CHUNK_TEXT","TA_SYSTEM_CHUNK_COORDINATES"],"title_field":"TA_SYSTEM_REFERENCE","filepath_field":"TA_SYSTEM_CAPTURE_DOC_ID","url_field":"TA_SYSTEM_REFERENCE_ID","vector_fields":["TA_SYSTEM_CHUNK_VECTORS"]},"embedding_dependency":{"endpoint":"https://qqqqqqqq.openai.azure.com/openai/deployments/TextEmbeddings/embeddings","authentication":{"type":"api_key","key":"xxxxxxxxxxxxxxxx "},"type":"endpoint"},"semantic_configuration":"developmentindex1_semanticConf","strictness":3,"top_n_documents":5,"in_scope":false,"filter":""}}],"messages":[{"role":"user","content":"1+1"}],"seed":null,"stream":true,"stream_options":{"include_usage":true}}
Thanks in advance.
Chandra shekar.M
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 61%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
santoshkc 14,270 Reputation points Microsoft External Staff2025-04-08T12:11:03.13+00:00 When using the chat/completions API with
stream: trueand the data_sources parameter for RAG, the response does not include usage details (like token counts). This is a known limitation with the current API behaviour.The
stream_options: { "include_usage": true }parameter you're trying might not supported at this time, which is why you're seeing the validation error.At the moment, the only way to get usage information is to disable streaming by setting
"stream": false. We understand that streaming responses are useful for better user experience, but unfortunately, usage metrics aren't available alongside it when RAG (data_sources) is enabled.We’re keeping an eye out for future updates from the Azure OpenAI team that might support this, and we'll be sure to share any changes once available.
I hope you understand! Thank you.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 22%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
SriLakshmi C 4,780 Reputation points Microsoft External Staff2025-04-09T09:53:14.0366667+00:00 Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Thank you!
-
Chandrashekar Machipeddi 0 Reputation points
2025-04-09T10:01:33.92+00:00 Hi, Santhosh, Srilakshmi.
Thank you for your quick update on this. We cannot turn of the stream as the response need to be stream only, we need to find another way to calculate the token count usage based on the input request and response, We have also raised the same issue with Microsoft support team, we are waiting for the reply from them as well.
Regards
Chandra shekar
-
santoshkc 14,270 Reputation points Microsoft External Staff
2025-04-10T10:26:06.68+00:00 Since streaming responses are essential for your application and usage details are not currently returned when
stream: trueis used along withdata_sources, a feasible alternative is to estimate token usage manually. This can help you monitor and approximate consumption for planning and cost-awareness purposes.In a typical conversation, both user input and AI responses contribute to token usage. On average, a person speaks around 125–150 words per minute, which equates to roughly 185–220 tokens. If the AI generates responses of similar length, output tokens are typically around 200–250 per minute. Taking into account the natural back-and-forth flow and conversation context, the estimated total token usage falls in the range of 400–500 tokens per minute.
I hope you understand! Thank you.
Sign in to comment
Sign in to answer