Azure OpenAI "on your data": stream_options parameter causes "Extra inputs are not permitted" validation error

Question

I'm trying to implement streaming with token usage tracking for Azure OpenAI "on your data" feature, but encountering a validation error when using the stream_options parameter that works fine with regular chat completions.

Environment Details

• Service: Azure OpenAI Service
• API Version: 2024-06-01 (also tested with 2024-10-21)
• Model: GPT-4o (also tested with GPT-4.1)
• Feature: Azure OpenAI "on your data" with Azure AI Search
• SDK: openai-python library (latest version)
• Authentication: System-assigned managed identity

What I'm Trying to Achieve

Enable streaming responses with token usage tracking when using Azure OpenAI "on your data" feature, similar to how stream_options: {"include_usage": true} works with regular chat completions.

Code That Causes the Error

from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

# Client setup
token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), 
    "https://cognitiveservices.azure.com/.default"
)

client = AzureOpenAI(
    azure_endpoint=endpoint,
    azure_ad_token_provider=token_provider,
    api_version="2024-06-01"
)

# This code fails with validation error
stream = client.chat.completions.create(
    model=deployment_name,
    messages=[{"role": "user", "content": "Test message"}],
    stream=True,
    stream_options={"include_usage": True},  # This line causes the error
    data_sources=[
        {
            "type": "azure_search",
            "parameters": {
                "endpoint": search_endpoint,
                "index_name": search_index,
                "authentication": {"type": "system_assigned_managed_identity"}
            }
        }
    ]
)

Error Message

{
  "error_message": "Validation error at #/stream_options: Extra inputs are not permitted",
  "content_filter_result": {}
}

What Works

1. Regular streaming without token usage (removing stream_options parameter):

# This works fine
stream = client.chat.completions.create(
    model=deployment_name,
    messages=[{"role": "user", "content": "Test message"}],
    stream=True,
    # stream_options removed
    data_sources=[...]
)

1. Token usage with regular chat completions (without data_sources):

# This also works fine
stream = client.chat.completions.create(
    model=deployment_name,
    messages=[{"role": "user", "content": "Test message"}],
    stream=True,
    stream_options={"include_usage": True}
    # No data_sources parameter
)

Is this a known limitation? Does Azure OpenAI "on your data" officially not support stream_options parameter?

Roadmap for support? Are there plans to add stream_options support for the "on your data" feature?

Official workaround? What's the recommended approach for tracking token usage when using "on your data" with streaming?

API documentation clarity? Should this limitation be explicitly documented in the "on your data" API reference?

Current Workaround

I'm currently using tiktoken for token estimation as suggested in the documentation:

import tiktoken

tokenizer = tiktoken.get_encoding("gpt2")

def estimate_tokens(text: str) -> int:
    return len(tokenizer.encode(text))

# Estimate input and output tokens manually
input_tokens = estimate_tokens(user_input)
output_tokens = estimate_tokens(response_text)

Impact

This limitation affects:

• Cost tracking for applications using "on your data"
• Usage monitoring and billing accuracy
• Rate limiting implementations
• Performance optimization based on token usage

Request

Could the Microsoft team please:

1. Confirm whether this is intended behavior or a bug
2. Provide timeline for potential stream_options support with "on your data"
3. Update documentation to clearly state this limitation
4. Suggest best practices for token tracking in this scenario

Thank you for any guidance or clarification the team can provide!

Answer 1

Hi @Kaja Sherief,

The stream_options parameter is currently only supported with standard chat completions and not when using the data_sources parameter with the Azure OpenAI "on your data" feature. When both stream_options and data_sources are included in the request, it results in a validation error stating "Extra inputs are not permitted." This is expected behavior, as the "on your data" pipeline does not yet support stream_options, even though basic streaming with stream: true is allowed.

At this time, the recommended approach for tracking token usage in "on your data" scenarios is to manually estimate tokens using libraries like tiktoken. Microsoft might plan to add support for this option in the future as the feature evolves. However, we appreciate your feedback and encourage you to submit a feature request through the Azure Feedback Forum: Post idea · Community (azure.com).

I hope you understand. And, if you have any further query do let us know.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful.