Evaluate recorded voice conversations with the Microsoft Foundry SDK

Use the Microsoft Foundry SDK to evaluate existing voice conversation recordings. Audio evaluators assess each complete conversation for qualities such as task completion and coherence.

You can also select audio evaluators when you run evaluations from the Foundry portal.

Audio evaluators grade the recording itself, so a transcript isn't required. You can grade recordings from these sources:

Prerequisites

  • Complete the cloud evaluation prerequisites and client setup.
  • A recording of each complete conversation.
  • An Azure OpenAI deployment of a gpt-realtime family model to act as the judge. Audio evaluators accept only these models. Set AZURE_AI_MODEL_DEPLOYMENT_NAME to this deployment.

The Python, C#, and JavaScript/TypeScript examples use the SDK client and judge deployment configured in Set up the SDK client.

Prepare the dataset

The evaluation reads a JSONL dataset that points to your recordings.

  1. Put the recordings in a local folder. Each file contains one complete conversation in the supported recording format. For example:

    recordings/
    └── calls/
        ├── order-status.wav
        └── billing-question.wav
    
  2. Register the folder as a versioned folder dataset in the same project. In Python, you can use project_client.datasets.upload_folder. Keep the returned dataset ID.

  3. Create a JSONL file with one row per recording, using the audio conversation schema. In each row, set conversation_artifacts.audio.asset_id to the folder dataset ID from the previous step and asset_file_path to the recording's path within that folder. For the preceding folder:

    {"conversation_artifacts":{"audio":{"asset_id":"<folder-dataset-id>","asset_file_path":"calls/order-status.wav"}}}
    {"conversation_artifacts":{"audio":{"asset_id":"<folder-dataset-id>","asset_file_path":"calls/billing-question.wav"}}}
    
  4. Upload the JSONL file as a separate dataset. Its ID is the data source for the evaluation run.

Define the data schema and evaluator

Define a data schema that requires conversation_artifacts, and add an audio evaluator with your judge deployment. Audio evaluators read conversation_artifacts.audio automatically, so omit data_mapping and audio_path overrides. For other audio evaluators and their scores, see Audio evaluators.

from azure.ai.projects.models import TestingCriterionAzureAIEvaluator
from openai.types.eval_create_params import DataSourceConfigCustom

data_source_config = DataSourceConfigCustom(
    type="custom",
    item_schema={
        "type": "object",
        "properties": {
            "conversation_artifacts": {
                "type": "object",
                "properties": {"audio": {"type": "object"}},
                "required": ["audio"],
            }
        },
        "required": ["conversation_artifacts"],
    },
    include_sample_schema=False,
)
testing_criteria = [
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="task_completion_audio",
        evaluator_name="builtin.task_completion_audio",
        evaluator_version="1",
        initialization_parameters={"deployment_name": model_deployment_name},
    )
]

Reference: TestingCriterionAzureAIEvaluator

Create the evaluation and run

Follow Create evaluation and run with this schema and evaluator. Skip the sample download and upload, and use your JSONL dataset ID as the run's data source. Give the run a name and set evaluation_level to conversation.

To poll for completion and interpret results, see Get cloud evaluation results. Check that each requested evaluator returned a numeric score for each recording. A score of zero is a valid task-completion result.

Next steps