Azure OpenAI GPT 3.5 not finetuned for tool calling? Not returning well formatted strings

Question

I believe Azure Open AI's gpt 3.5 model is not finetuned for tool calling to return properly formatted JSONs.

Today I tried to migrate a large service (~200k tokens/minute) from OpenAI to Azure's API. We are using function calling, GPT 3.5 turbo latest model, latest api version, latest openai package version. All of my code is exactly identical, except my client instantiation. I am using a GPT 3.5 latest model in my deployment.

if DEPLOYMENT == "openai":
    client = Client()
    gpt_3_5 = "gpt-3.5-turbo-0125"
    gpt_4 = "gpt-4-turbo-preview"
elif DEPLOYMENT == "azure":
    client = AzureOpenAI(
        api_key=os.getenv("AZURE_OPENAI_API_KEY"),
        api_version="2024-02-01",
        azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
    )
    gpt_3_5 = "gpt35_more_requests"
    gpt_4 = None

The function in question is standard, it simply calls the chat completions api to request a category for a product from its string. Previously, this worked consistently with OpenAI's API. When I switch to Azure's, I get ~0% of calls working. Responses typically come back as "arguments": "{\n category: "furniture"\n}", or simply with no category at all.

I have tried changing the prompt to give specific instructions. But fundamentally, I just want the same model as OpenAI's. Is this possible?

I believe either the system prompt is different, or the underlying model is different. Is it possible to receive access to a model fine-tuned for function calling?

@backoff.on_exception(backoff.expo, Exception, max_tries=1, max_time=5)
def predict_category(description, categories):
    response = client.chat.completions.create(
        model=gpt_3_5,
        temperature=0,
        tools=[
            {
                "type": "function",
                "function": {
                    "name": "classify_item",
                    "description": PROMPTS["predict_category_system"][-1],
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "category": {"type": "string", "enum": categories}
                        },
                        "required": ["category"],
                    },
                },
            }
        ],
        tool_choice={"type": "function", "function": {"name": "classify_item"}},
        messages=[
            {"role": "system", "content": PROMPTS["predict_category_message"][-1]},
            {"role": "user", "content": description},
        ],
    )
    cost = get_gpt35_cost(response)
    return (
        json.loads(response.choices[0].message.tool_calls[0].function.arguments).get(
            "category"
        ),
        cost,
    )

Answer 1

@Baptiste Cumin

Thanks for reaching out to us, I think this issue is related to the version of the API and differences in the implementation of Azure's OpenAI API versus OpenAI's API, or possibly due to differences in the models themselves.

Here are a few things you could try:

1. Verify the model: Make sure the models you're using in both Azure and OpenAI are identical. You mentioned using "gpt-3.5-turbo-0125" for OpenAI and "gpt35_more_requests" for Azure. If these are not the same model, it could explain the discrepancy. Please share more details for which two models you are comparing.
2. Check the API version: You may want to ensure that the API versions you are using for both OpenAI and Azure are compatible and support the features you're trying to use.
3. Debug the function: Try to isolate the problem by testing the function with different inputs and settings. For example, you could try using a simpler prompt or different categories to see if the problem persists.

If none of the above steps resolve the issue, it might be best to reach out to Azure support for help. Please let me know if you have no support plan, I am happy to enable you a free ticket for this case. They could provide more specific guidance based on their knowledge of the models and APIs.

I hope this helps.

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.

Answer 2

Thank you for your answer Yutong!

Here' what I tried:

1. I verified the model, both point to the latest GPT 3.5 instances available. My openai deployment points to: model name: gpt-35-turboModel version: 0301
2. The api version is Feb 2024 (see the linked code above)
3. I've tried debugging the function with different input and settings, to no avail. But fundamentally as I say, I would prefer to be able to move more seamlessly from Open AI's client to Azure's, as having to rewrite prompts for all my application will be lengthy.

I do not have the support plan but I would appreciate a ticket.