How do I evaluate reasoning models (o3-mini) using Azure AI Foundry evaluation or Prompt Flow?

Question

How do I evaluate reasoning models (o3-mini) using Azure AI Foundry evaluation or Prompt Flow?

Kavishka Gamage 0

I have tried to evaluate the o3-mini model using an existing dataset, via Azure Machine Learning Workspace PromptFlow, as well as in Azure AI Foundry Evaluation, Azure OpenAI Evaluation, and PromptFlow options. However, it failed due to the parameter differences between GPT models and reasoning models.

What are alternative ways to evaluate the o3-mini model to benchmark it against an existing dataset?

 Error code: 400 - {'error': {'message': "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}

Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-20T13:55:16.41+00:00
Hi Kavishka Gamage

Hope the below answer from Amira Bedhiafi helped debugging the issue

Attached relevant documentation for reference.

Evaluate through Portal

Evaluate through SDK

Run evaluation online

Thank you.
Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-21T08:36:52.2766667+00:00

Hi Kavishka Gamage

We have not heard from you. Hope the pointer shared were useful to you.

Thank you.
Kavishka Gamage 0 Reputation points

2025-03-21T08:43:33.93+00:00

Hi Manas,

I am not clear on the direction, so an alternative to using the UI on Azure AI Foundry or Machine Learning Studio would be to use the Azure AI SDK. Is that what you're suggesting?

Is there a way to test the reasoning model in the PromptFlow UI itself? There doesn't seem to be an option to change the parameters there
Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-21T11:25:57.2633333+00:00

Hi Kavishka Gamage

We have to create a prompt flow for 03 using SDK then use evaluation (groundess) from SDK or UI from evaluate our flow.

Reference

Sample RAG add your own data using max_completion_tokens instead of max_tokens

Attached sample document on evaluation from portal .

Thank you
Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-24T08:19:38.7133333+00:00

Hi Kavishka Gamage

We have not heard from you. Hope the pointers shared were useful to you.

Thank you.
Kavishka Gamage 0 Reputation points

2025-03-24T09:41:19.02+00:00

Yes, Thank you. I haven't found an exact solution yet.

I have the requirement to evaluate dataset with multiple metrics like accuracy, F1-score, GPT-similarity, ada-similarity etc. I find it's difficult to do for reasoning models.

But thank you for all the help.

1 answer

Your answer

Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-20T13:55:16.41+00:00

Hi Kavishka Gamage

Hope the below answer from Amira Bedhiafi helped debugging the issue

Attached relevant documentation for reference.

Evaluate through Portal

Evaluate through SDK

Run evaluation online

Thank you.
Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-21T08:36:52.2766667+00:00

Hi Kavishka Gamage

We have not heard from you. Hope the pointer shared were useful to you.

Thank you.
Kavishka Gamage 0 Reputation points

2025-03-21T08:43:33.93+00:00

Hi Manas,

I am not clear on the direction, so an alternative to using the UI on Azure AI Foundry or Machine Learning Studio would be to use the Azure AI SDK. Is that what you're suggesting?

Is there a way to test the reasoning model in the PromptFlow UI itself? There doesn't seem to be an option to change the parameters there
Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-21T11:25:57.2633333+00:00

Hi Kavishka Gamage

We have to create a prompt flow for 03 using SDK then use evaluation (groundess) from SDK or UI from evaluate our flow.

Reference

Sample RAG add your own data using max_completion_tokens instead of max_tokens

Attached sample document on evaluation from portal .

Thank you
Manas Mohanty 17,185 Reputation points Microsoft External Staff Moderator

2025-03-24T08:19:38.7133333+00:00

Hi Kavishka Gamage

We have not heard from you. Hope the pointers shared were useful to you.

Thank you.
Kavishka Gamage 0 Reputation points

2025-03-24T09:41:19.02+00:00

Yes, Thank you. I haven't found an exact solution yet.

I have the requirement to evaluate dataset with multiple metrics like accuracy, F1-score, GPT-similarity, ada-similarity etc. I find it's difficult to do for reasoning models.

But thank you for all the help.

Answer 1

Hello Kavishka !

Thank you for posting on Microsoft Learn.

Unlike standard GPT models, reasoning models like o3-mini use:

max_completion_tokens instead of max_tokens
temperature (for randomness control)
top_p (nucleus sampling)

If you are using Prompt Flow within Azure AI Studio, your YAML or JSON payload should include the parameter:

parameters:
  model: "o3-mini"
  max_completion_tokens: 512
  temperature: 0.7
  top_p: 0.95
  prompt: "Evaluate the following reasoning dataset..."

For Azure AI Foundry Evaluation, if you are running evaluations via API or SDK, update your payload:

{
  "model": "o3-mini",
  "max_completion_tokens": 512,
  "temperature": 0.7,
  "top_p": 0.95,
  "input": "Evaluate the reasoning ability of this dataset..."
}

Share via

How do I evaluate reasoning models (o3-mini) using Azure AI Foundry evaluation or Prompt Flow?

1 answer

Your answer