Evaluating an agent creating more than the "maximum_conversations" allowed.

Question

Hello, i hope you are all doing well.

I was evaluating an agent and i set the "maximum conversations" to 1 per test case.

The agent generated more than 100 conversations per case, with different ID's.

I have attached a screenshot regarding the issue.

Can someone help me with this topic?

Thank you

Screenshot 2026-08-26 101305.png

Answer 1

Welcome to Microsoft Q&A.

@André Silveira I hope you are doing well.

Looking at the screenshot, the key setting is:

num_conversations: 1
max_turns: 6

num_conversations: 1 means one conversation per test case, not one conversation for the entire evaluation.

So, if your dataset contains 100 test cases, the evaluation can generate around 100 different conversation IDs even though num_conversations is set to 1.

max_turns: 6 is separate — it controls how many turns each generated conversation can contain.

I would first check how many rows/test cases are in the dataset used for the evaluation. If there is only one test case and you are still getting 100+ conversations, then that would be unexpected and worth investigating further.

Microsoft documentation:

https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app

https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/cloud-evaluation

If this answer helped clarify the platform capabilities and save you troubleshooting time, please consider marking it as Accepted. This helps others in the community find similar solutions.