Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Use the Microsoft Foundry SDK to evaluate existing voice conversation recordings. Audio evaluators assess each complete conversation for qualities such as task completion and coherence.
You can also select audio evaluators when you run evaluations from the Foundry portal.
Audio evaluators grade the recording itself, so a transcript isn't required. You can grade recordings from these sources:
- Existing recordings: Follow this article.
- Simulated conversations: Simulate voice conversations with your agent, and evaluate the generated recordings in the same run.
- Deployed voice agent conversations: Evaluate conversations from Application Insights traces by trace or conversation ID.
Prerequisites
- Complete the cloud evaluation prerequisites and client setup.
- A recording of each complete conversation.
- An Azure OpenAI deployment of a
gpt-realtimefamily model to act as the judge. Audio evaluators accept only these models. SetAZURE_AI_MODEL_DEPLOYMENT_NAMEto this deployment.
The Python, C#, and JavaScript/TypeScript examples use the SDK client and judge deployment configured in Set up the SDK client.
Prepare the dataset
The evaluation reads a JSONL dataset that points to your recordings.
Put the recordings in a local folder. Each file contains one complete conversation in the supported recording format. For example:
recordings/ └── calls/ ├── order-status.wav └── billing-question.wavRegister the folder as a versioned folder dataset in the same project. In Python, you can use
project_client.datasets.upload_folder. Keep the returned dataset ID.Create a JSONL file with one row per recording, using the audio conversation schema. In each row, set
conversation_artifacts.audio.asset_idto the folder dataset ID from the previous step andasset_file_pathto the recording's path within that folder. For the preceding folder:{"conversation_artifacts":{"audio":{"asset_id":"<folder-dataset-id>","asset_file_path":"calls/order-status.wav"}}} {"conversation_artifacts":{"audio":{"asset_id":"<folder-dataset-id>","asset_file_path":"calls/billing-question.wav"}}}Upload the JSONL file as a separate dataset. Its ID is the data source for the evaluation run.
Define the data schema and evaluator
Define a data schema that requires conversation_artifacts, and add an audio evaluator with your judge deployment. Audio evaluators read conversation_artifacts.audio automatically, so omit data_mapping and audio_path overrides. For other audio evaluators and their scores, see Audio evaluators.
from azure.ai.projects.models import TestingCriterionAzureAIEvaluator
from openai.types.eval_create_params import DataSourceConfigCustom
data_source_config = DataSourceConfigCustom(
type="custom",
item_schema={
"type": "object",
"properties": {
"conversation_artifacts": {
"type": "object",
"properties": {"audio": {"type": "object"}},
"required": ["audio"],
}
},
"required": ["conversation_artifacts"],
},
include_sample_schema=False,
)
testing_criteria = [
TestingCriterionAzureAIEvaluator(
type="azure_ai_evaluator",
name="task_completion_audio",
evaluator_name="builtin.task_completion_audio",
evaluator_version="1",
initialization_parameters={"deployment_name": model_deployment_name},
)
]
Reference: TestingCriterionAzureAIEvaluator
Create the evaluation and run
Follow Create evaluation and run with this schema and evaluator. Skip the sample download and upload, and use your JSONL dataset ID as the run's data source. Give the run a name and set evaluation_level to conversation.
To poll for completion and interpret results, see Get cloud evaluation results. Check that each requested evaluator returned a numeric score for each recording. A score of zero is a valid task-completion result.
Next steps
- To generate voice conversations and evaluate them, see Simulate conversations.
- To review other voice evaluation workflows, see Voice agent tracing, monitoring, and evaluation.
- If your project uses network isolation, see Configure virtual network support for evaluation.