Does streaming work with tool_calls yet on AzureOpenAI version 2023-12-01-preview? I'm getting an empty choices array.

Darweenist 5

I just migrated to the newest OpenAI version which uses tool calls instead of function calls. I set stream = true. Just got it to work with OpenAI, but upon migrating to AzureOpenAI and not changing any code in my tool calls handler, I am getting IndexError: list index out of range on the line delta = res.choices[0].delta :

response_gen

async for res in response_gen:
    delta = res.choices[0].delta

I've seen some issues with AzureOpenAI returning an empty first spot in the array in earlier preview versions. Anyone know what's the issue here? I'm on 2023-12-01-preview, using GPT-4 1106-Preview.

AshokPeddakotla-MSFT 32,946 Reputation points

2024-02-16T06:10:47.1833333+00:00
Darweenist Greetings & Welcome to Microsoft Q&A forum!

Supported models are

gpt-35-turbo (1106)

gpt-4 (1106-preview)

The error message "IndexError: list index out of range" usually occurs when you are trying to access an index that does not exist in a list. In your case, it seems like the res.choices list is empty, which is why you are getting this error.

One possible reason for this could be that the response from the Azure OpenAI service is not in the expected format. It is possible that the response is empty or does not contain the expected data.

To debug this issue, you can try printing the response from the Azure OpenAI service and check if it contains the expected data.

response_gen async for res in response_gen: try: delta = res.choices[0].delta except IndexError: print("Error: Response from Azure OpenAI service is empty.")

Regarding the streaming, I will check and confirm.

I hope this helps! Let me know if you have any further questions.
Darweenist 5 Reputation points

2024-02-16T08:19:41.5933333+00:00

Thanks. Please let me know if streaming is supported. Will keep debugging on my end.
AshokPeddakotla-MSFT 32,946 Reputation points

2024-02-16T09:52:50.9533333+00:00

Darweenist Can you also share complete error logs or network trace and the region where your model is deployed?

Darweenist 5

Here is the relevant code:


kwargs = {
        "model": request.model.value,
        "messages": messages,
        "temperature": request.temperature,
        **(request.kwargs or {}),
    }

    # cannot do function calls with streaming, so don't call Azure with function calls for now

    response = await azureaclient.chat.completions.create(**kwargs)

    async for res in response:
        delta = res.choices[0].delta
        if delta and delta.tool_calls:
            logger.info("found tool calls")
            tcchunklist = delta.tool_calls
            for tcchunk in tcchunklist:
                # Additional logging to see the entire tool call
                logger.info(f"Tool call chunk: {tcchunk}")

                # Try-except block to catch any issues with missing or None arguments
                try:
                    if tcchunk.function.arguments:
                        arguments_str = json.dumps(tcchunk.function.arguments)
                    else:
                        arguments_str = "No arguments"
                    logger.info(f"handling tool call with name: {tcchunk.function.name} and arguments: {arguments_str}")
                except Exception as e:
                    logger.error(f"Error handling tool call arguments: {e}")
        if res.choices[0].finish_reason:
            logger.info(f"finish_reason: {res.choices[0].finish_reason}")
        else:
            logger.info("no finish reason")

Here is the error trace. You should be able to see where the error occurs: Screenshot 2024-02-16 at 3.36.01 PM

The exact same code works for OpenAI, but doesn't for AzureOpenAI. I'm using streaming and tool calls. Something about this combination doesn't work for AzureOpenAI's most recent preview version. AzureOpenAI works totally as expected for me when it comes to any other GPT-4 call (i.e. streaming without tool-calls, regular non-streaming chat completions, etc.). Please let me know if you spot something I'm doing wrong or if it's a problem with AzureOpenAI.

1 answer

Evgeny Minkevich 0 Reputation points

2024-03-14T18:45:29.5+00:00
By default Azure OpenAI has content filtering on [1].

So the first chunk might come with empty "choices". If you skip the chunks with empty "choices" it should work.

Consider putting something along these lines into your code:

if not res.choices: continue

[1] Azure OpenAI Content Filtering
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Does streaming work with tool_calls yet on AzureOpenAI version 2023-12-01-preview? I'm getting an empty choices array.

1 answer

Your answer