Creates a run of the specified evaluation against the provided data source.

Evaluation runs are processed asynchronously. Use the returned run identifier to retrieve the run and monitor its status.

The identifier of the evaluation to run.

The name of the run.

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Defines how the service obtains the data to evaluate. Choose the variant that matches one of these workflows: - Evaluate existing data: Provide a dataset or inline items containing query, response, and optional ground-truth fields for single-turn evaluation, or complete conversations for multi-turn evaluation. The service evaluates the supplied responses without invoking a target. - Evaluate collected interactions: Select traces or stored responses from previous model or agent interactions. The service retrieves the interactions and evaluates them. - Invoke a target with supplied inputs: Provide queries for single-turn evaluation or scenarios for multi-turn conversation simulation, and specify a model or agent target. The service invokes the target, captures its responses, and evaluates them. - Generate inputs and invoke a target: Provide generation instructions instead of individual queries or scenarios. The service generates the inputs, invokes the specified target, captures its responses, and evaluates them. The selected source must conform to the evaluation's data_source_config.

The type of data source. Always jsonl.

Determines what populates the item namespace in the data source.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The type of run data source. Always completions.

Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

The type of response format being defined. Always text.

The type of response format being defined. Always json_schema.

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

The type of response format being defined. Always json_object.

The name of the model to use for generating completions (e.g. "o3-mini").

Determines what populates the item namespace in this run's data source.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The type of source. Always stored_completions.

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

The type of run data source. Always responses.

Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

An object specifying the format that the model must output. Configuring { "type": "json_schema" } enables Structured Outputs, which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide. The default format is { "type": "text" } with no additional options. Not recommended for gpt-4o and newer models:* Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

The type of response format being defined. Always json_schema.

A description of what the response format is for, used by the model to determine how to respond in the format.

The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

Default response format. Used to generate text responses.

The type of response format being defined. Always text.

JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

The type of response format being defined. Always json_object.

The name of the model to use for generating completions (e.g. "o3-mini").

Determines what populates the item namespace in this run's data source.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The type of run data source. Always responses.

Optional reasoning effort parameter. This is a query parameter used to select responses.

A data source for evaluation runs over agent traces stored in Application Insights. See create an evaluation dataset from traces.

The type of data source, always azure_ai_traces.

Collection of Agent trace identifiers that should be evaluated.

Collection of conversation identifiers to filter traces by conversation.

The agent ID used to filter traces for evaluation.

The agent name used to filter traces for evaluation.

Lookback window (in hours) applied when retrieving traces from Application Insights. For scheduled evaluations this is inferred from the recurrence interval.

Sampling limit applied to traces retrieved for evaluation.

The delay to apply for ingestion when querying traces.

A data source that generates synthetic data for an evaluation run. See generate synthetic data for cloud evaluation and generate a synthetic evaluation dataset.

The type of data source, always azure_ai_synthetic_data_gen.

Represents the parameters for synthetic data generation item generation.

The type of item generation parameters.

The maximum number of data samples to generate.

The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.

The name of the model deployment to use for generating synthetic data.

The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.

The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.

The optional seed data content source files for data generation.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Generates seed test-case scenarios via the data generation endpoint and then evaluates a target by simulating multi-turn user conversations over those scenarios. The generation_sources in synthetic_data_generation_configuration are forwarded to the data generation backend to produce seed test cases. Each generated scenario is then simulated against target: the model in model_configuration plays the simulated user and drives the conversation according to default_simulation_configuration until it reaches the desired length or maximum turn limit. The evaluation's testing criteria are then applied to the generated conversations.

The type of data source, always azure_ai_synthetic_data_generation_with_simulation.

Seed data-generation configuration for SyntheticDataGenerationWithSimulationEvalRunDataSource. The generation_sources are forwarded directly to the data generation endpoint to produce the seed test cases that are subsequently simulated.

Number of seed test cases to generate.

Name of the dataset where the generated seed test cases are stored. When omitted, the service generates a name automatically.

Seed generation sources (prompt, agent, or file) forwarded to the data generation endpoint. Multiple sources may be provided.

Configures the model that plays the simulated user. This model is separate from the evaluated target.

Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.

Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.

The voice kind. Always azure-standard.

The Azure neural voice name.

The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.

Configures default conversation behavior and optional run-wide persistence of generated conversations.

Hard limit on turns in each conversation. When omitted, the service defaults to 20.

Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.

Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.

Configures effects applied to simulated conversation audio.

Effects to apply to simulated audio.

Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.

Configures conversation behavior for the simulated user.

Configuration for simulated user interruption behavior.

Configures the default interruption behavior for the simulated user.

The interruption type, always default.

Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.

Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

A data source that retrieves deployed interactions for continuous evaluation. See evaluate deployed interactions.

The type of data source, always azure_ai_responses.

Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.

The type of item generation parameters, always response_retrieval.

The maximum number of turns of chat history to evaluate.

Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.

The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Maximum number of evaluation runs allowed per hour.

The event configuration name associated with this evaluation run.

A data source for evaluating model or agent target completions. See evaluate conversations.

The type of data source, always azure_ai_target_completions.

The source configuration for inline or file data.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Configures how the service prepares input items for an evaluation run before applying testing criteria. Depending on type, preparation can generate new inputs, retrieve existing responses, transform source data, or simulate conversations. Each type selects a workflow-specific model with the parameters required by that preparation workflow. These parameters do not define the evaluated target or the testing criteria.

Represents the parameters for red team item generation.

The type of item generation parameters.

The collection of attack strategies to be used.

The number of turns allowed in the game.

Represents the parameters for red team seed prompts item generation.

The type of item generation parameters, always red_team_seed_prompts.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Represents the parameters for red team taxonomy item generation.

The type of item generation parameters, always red_team_taxonomy.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.

The type of item generation parameters, always response_retrieval.

The maximum number of turns of chat history to evaluate.

Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.

The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Parameters for generating target completions for simulated conversations. See simulate conversations.

The type of item generation parameters, always target_completion.

The maximum number of target completions to execute concurrently. The services defaults to 1 if a value is not specified by the caller, which invokes the target sequentially.

Represents the parameters for synthetic data generation item generation.

The type of item generation parameters.

The maximum number of data samples to generate.

The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.

The name of the model deployment to use for generating synthetic data.

The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.

The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.

The optional seed data content source files for data generation.

A CSV data source for evaluation runs. See evaluation datasets.

The type of data source, always csv.

Represents a reference to an uploaded CSV file used as a source for evaluation data.

The type of source, always file_id.

The identifier of the uploaded CSV file.

Represents a data source for red team evaluation runs.

The type of data source. Always azure_ai_red_team.

Configures how the service prepares input items for an evaluation run before applying testing criteria. Depending on type, preparation can generate new inputs, retrieve existing responses, transform source data, or simulate conversations. Each type selects a workflow-specific model with the parameters required by that preparation workflow. These parameters do not define the evaluated target or the testing criteria.

Represents the parameters for red team item generation.

The type of item generation parameters.

The collection of attack strategies to be used.

The number of turns allowed in the game.

Represents the parameters for red team seed prompts item generation.

The type of item generation parameters, always red_team_seed_prompts.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Represents the parameters for red team taxonomy item generation.

The type of item generation parameters, always red_team_taxonomy.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.

The type of item generation parameters, always response_retrieval.

The maximum number of turns of chat history to evaluate.

Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.

The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Parameters for generating target completions for simulated conversations. See simulate conversations.

The type of item generation parameters, always target_completion.

The maximum number of target completions to execute concurrently. The services defaults to 1 if a value is not specified by the caller, which invokes the target sequentially.

Represents the parameters for synthetic data generation item generation.

The type of item generation parameters.

The maximum number of data samples to generate.

The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.

The name of the model deployment to use for generating synthetic data.

The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.

The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.

The optional seed data content source files for data generation.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

A data source that selects traces for evaluation. See create an evaluation dataset from traces.

The type of data source, always azure_ai_trace_data_source.

Base class for polymorphic trace sources nested within an AzureAITraceDataSourceEvalRunDataSource.

A trace source that selects traces by explicit trace IDs.

The type of trace source, always trace_id_source.

Collection of trace identifiers to evaluate.

A trace source that selects traces by conversation IDs.

The type of trace source, always conversation_id_source.

Collection of conversation identifiers to filter traces by.

A trace source that selects traces by agent reference with time-based filtering.

The type of trace source, always agent_filter.

The identifier of the agent to filter traces by.

The name of the agent to filter traces by.

The version of the agent to filter traces by.

Maximum number of traces to sample. Must be greater than 0.

Specifies the type of filter strategy used to select traces for evaluation.

Represents a data source for benchmark evaluation runs.

The type of data source, always azure_ai_benchmark_preview.

The target model or agent to evaluate against the benchmark. When using azure_ai_model target, sampling_params must not be provided; inference parameters are auto-filled from the benchmark specification stored in eval group properties.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Evaluates a model or agent by simulating user conversations against it. Each source item defines a test scenario. Use a JSONL source or an inline_user_conversation_simulation source with strongly typed test cases. JSONL attributes use the inline test-case property names by default; specify data_mapping only when the dataset uses different names. The model specified by model_configuration generates the simulated user's turns and drives the conversation with target until it reaches the desired length or maximum turn limit. The evaluation's testing criteria are then applied to the conversation.

The type of data source, always azure_ai_user_conversation_simulation.

Sources supported by user conversation simulation.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The source type. Always inline_user_conversation_simulation.

Test scenarios from which user conversations are simulated. At least one test case is required.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Configures the model that plays the simulated user. This model is separate from the evaluated target.

Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.

Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.

The voice kind. Always azure-standard.

The Azure neural voice name.

The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.

Configures default conversation behavior and optional run-wide persistence of generated conversations.

Hard limit on turns in each conversation. When omitted, the service defaults to 20.

Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.

Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.

Configures effects applied to simulated conversation audio.

Effects to apply to simulated audio.

Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.

Configures conversation behavior for the simulated user.

Configuration for simulated user interruption behavior.

Configures the default interruption behavior for the simulated user.

The interruption type, always default.

Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.

Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.

Maps JSONL dataset attributes to user conversation simulation inputs when their names differ from the defaults. The default attribute names are test_case_id, test_case_category, test_case_description, and simulation_configuration, matching UserConversationSimulationTestCase. Omit when the dataset uses these names or when using inline_user_conversation_simulation.

Set of immutable 16 key-value pairs that can be attached to an object for storing additional information. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

The level at which evaluation is performed.

The type of the object. Always "eval.run".

Unique identifier for the evaluation run.

The identifier of the associated evaluation.

The status of the evaluation run.

The model that is evaluated, if applicable.

The name of the evaluation run.

Unix timestamp (in seconds) when the evaluation run was created.

The URL to the rendered evaluation run report on the UI dashboard.

Usage statistics for each model during the evaluation run.

Results per testing criteria applied during the evaluation run.

Information about the run's data source.

The type of data source. Always jsonl.

Determines what populates the item namespace in the data source.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The type of run data source. Always completions.

Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

The type of response format being defined. Always text.

The type of response format being defined. Always json_schema.

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

The type of response format being defined. Always json_object.

The name of the model to use for generating completions (e.g. "o3-mini").

Determines what populates the item namespace in this run's data source.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The type of source. Always stored_completions.

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

The type of run data source. Always responses.

Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

An object specifying the format that the model must output. Configuring { "type": "json_schema" } enables Structured Outputs, which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide. The default format is { "type": "text" } with no additional options. Not recommended for gpt-4o and newer models:* Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

The type of response format being defined. Always json_schema.

A description of what the response format is for, used by the model to determine how to respond in the format.

The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

Default response format. Used to generate text responses.

The type of response format being defined. Always text.

JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

The type of response format being defined. Always json_object.

The name of the model to use for generating completions (e.g. "o3-mini").

Determines what populates the item namespace in this run's data source.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The type of run data source. Always responses.

Optional reasoning effort parameter. This is a query parameter used to select responses.

A data source for evaluation runs over agent traces stored in Application Insights. See create an evaluation dataset from traces.

The type of data source, always azure_ai_traces.

Collection of Agent trace identifiers that should be evaluated.

Collection of conversation identifiers to filter traces by conversation.

The agent ID used to filter traces for evaluation.

The agent name used to filter traces for evaluation.

Lookback window (in hours) applied when retrieving traces from Application Insights. For scheduled evaluations this is inferred from the recurrence interval.

Sampling limit applied to traces retrieved for evaluation.

The delay to apply for ingestion when querying traces.

A data source that generates synthetic data for an evaluation run. See generate synthetic data for cloud evaluation and generate a synthetic evaluation dataset.

The type of data source, always azure_ai_synthetic_data_gen.

Represents the parameters for synthetic data generation item generation.

The type of item generation parameters.

The maximum number of data samples to generate.

The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.

The name of the model deployment to use for generating synthetic data.

The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.

The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.

The optional seed data content source files for data generation.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Generates seed test-case scenarios via the data generation endpoint and then evaluates a target by simulating multi-turn user conversations over those scenarios. The generation_sources in synthetic_data_generation_configuration are forwarded to the data generation backend to produce seed test cases. Each generated scenario is then simulated against target: the model in model_configuration plays the simulated user and drives the conversation according to default_simulation_configuration until it reaches the desired length or maximum turn limit. The evaluation's testing criteria are then applied to the generated conversations.

The type of data source, always azure_ai_synthetic_data_generation_with_simulation.

Seed data-generation configuration for SyntheticDataGenerationWithSimulationEvalRunDataSource. The generation_sources are forwarded directly to the data generation endpoint to produce the seed test cases that are subsequently simulated.

Number of seed test cases to generate.

Name of the dataset where the generated seed test cases are stored. When omitted, the service generates a name automatically.

Seed generation sources (prompt, agent, or file) forwarded to the data generation endpoint. Multiple sources may be provided.

Configures the model that plays the simulated user. This model is separate from the evaluated target.

Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.

Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.

The voice kind. Always azure-standard.

The Azure neural voice name.

The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.

Configures default conversation behavior and optional run-wide persistence of generated conversations.

Hard limit on turns in each conversation. When omitted, the service defaults to 20.

Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.

Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.

Configures effects applied to simulated conversation audio.

Effects to apply to simulated audio.

Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.

Configures conversation behavior for the simulated user.

Configuration for simulated user interruption behavior.

Configures the default interruption behavior for the simulated user.

The interruption type, always default.

Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.

Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

A data source that retrieves deployed interactions for continuous evaluation. See evaluate deployed interactions.

The type of data source, always azure_ai_responses.

Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.

The type of item generation parameters, always response_retrieval.

The maximum number of turns of chat history to evaluate.

Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.

The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Maximum number of evaluation runs allowed per hour.

The event configuration name associated with this evaluation run.

A data source for evaluating model or agent target completions. See evaluate conversations.

The type of data source, always azure_ai_target_completions.

The source configuration for inline or file data.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Configures how the service prepares input items for an evaluation run before applying testing criteria. Depending on type, preparation can generate new inputs, retrieve existing responses, transform source data, or simulate conversations. Each type selects a workflow-specific model with the parameters required by that preparation workflow. These parameters do not define the evaluated target or the testing criteria.

Represents the parameters for red team item generation.

The type of item generation parameters.

The collection of attack strategies to be used.

The number of turns allowed in the game.

Represents the parameters for red team seed prompts item generation.

The type of item generation parameters, always red_team_seed_prompts.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Represents the parameters for red team taxonomy item generation.

The type of item generation parameters, always red_team_taxonomy.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.

The type of item generation parameters, always response_retrieval.

The maximum number of turns of chat history to evaluate.

Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.

The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Parameters for generating target completions for simulated conversations. See simulate conversations.

The type of item generation parameters, always target_completion.

The maximum number of target completions to execute concurrently. The services defaults to 1 if a value is not specified by the caller, which invokes the target sequentially.

Represents the parameters for synthetic data generation item generation.

The type of item generation parameters.

The maximum number of data samples to generate.

The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.

The name of the model deployment to use for generating synthetic data.

The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.

The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.

The optional seed data content source files for data generation.

A CSV data source for evaluation runs. See evaluation datasets.

The type of data source, always csv.

Represents a reference to an uploaded CSV file used as a source for evaluation data.

The type of source, always file_id.

The identifier of the uploaded CSV file.

Represents a data source for red team evaluation runs.

The type of data source. Always azure_ai_red_team.

Configures how the service prepares input items for an evaluation run before applying testing criteria. Depending on type, preparation can generate new inputs, retrieve existing responses, transform source data, or simulate conversations. Each type selects a workflow-specific model with the parameters required by that preparation workflow. These parameters do not define the evaluated target or the testing criteria.

Represents the parameters for red team item generation.

The type of item generation parameters.

The collection of attack strategies to be used.

The number of turns allowed in the game.

Represents the parameters for red team seed prompts item generation.

The type of item generation parameters, always red_team_seed_prompts.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Represents the parameters for red team taxonomy item generation.

The type of item generation parameters, always red_team_taxonomy.

The collection of attack strategies to be used.

The number of turns allowed in the game.

The type of jsonl source. Always file_content.

The content of the jsonl file.

Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.

The type of item generation parameters, always response_retrieval.

The maximum number of turns of chat history to evaluate.

Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.

The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

Parameters for generating target completions for simulated conversations. See simulate conversations.

The type of item generation parameters, always target_completion.

The maximum number of target completions to execute concurrently. The services defaults to 1 if a value is not specified by the caller, which invokes the target sequentially.

Represents the parameters for synthetic data generation item generation.

The type of item generation parameters.

The maximum number of data samples to generate.

The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.

The name of the model deployment to use for generating synthetic data.

The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.

The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.

The optional seed data content source files for data generation.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

A data source that selects traces for evaluation. See create an evaluation dataset from traces.

The type of data source, always azure_ai_trace_data_source.

Base class for polymorphic trace sources nested within an AzureAITraceDataSourceEvalRunDataSource.

A trace source that selects traces by explicit trace IDs.

The type of trace source, always trace_id_source.

Collection of trace identifiers to evaluate.

A trace source that selects traces by conversation IDs.

The type of trace source, always conversation_id_source.

Collection of conversation identifiers to filter traces by.

A trace source that selects traces by agent reference with time-based filtering.

The type of trace source, always agent_filter.

The identifier of the agent to filter traces by.

The name of the agent to filter traces by.

The version of the agent to filter traces by.

Maximum number of traces to sample. Must be greater than 0.

Specifies the type of filter strategy used to select traces for evaluation.

Represents a data source for benchmark evaluation runs.

The type of data source, always azure_ai_benchmark_preview.

The target model or agent to evaluate against the benchmark. When using azure_ai_model target, sampling_params must not be provided; inference parameters are auto-filled from the benchmark specification stored in eval group properties.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Evaluates a model or agent by simulating user conversations against it. Each source item defines a test scenario. Use a JSONL source or an inline_user_conversation_simulation source with strongly typed test cases. JSONL attributes use the inline test-case property names by default; specify data_mapping only when the dataset uses different names. The model specified by model_configuration generates the simulated user's turns and drives the conversation with target until it reaches the desired length or maximum turn limit. The evaluation's testing criteria are then applied to the conversation.

The type of data source, always azure_ai_user_conversation_simulation.

Sources supported by user conversation simulation.

The type of jsonl source. Always file_content.

The content of the jsonl file.

The type of jsonl source. Always file_id.

The identifier of the file.

The source type. Always inline_user_conversation_simulation.

Test scenarios from which user conversations are simulated. At least one test case is required.

Base class for targets with discriminator support.

A target that identifies an Azure AI model. See evaluate administrator-connected models.

The type of target, always azure_ai_model.

The name of the model deployment to invoke as the evaluation target.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

A target that identifies an Azure AI agent. See evaluate model or agent targets.

The type of target, always azure_ai_agent.

The name of the Azure AI agent to invoke as the evaluation target.

The version of the Azure AI agent. When omitted, the latest version is used.

Optional descriptions of tools available to the agent.

Tool definitions made available when invoking the agent.

Configures the model that plays the simulated user. This model is separate from the evaluated target.

Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.

Represents a set of parameters used to control the sampling behavior of a language model during text generation.

The temperature parameter for sampling. Defaults to 1.0.

The top-p parameter for nucleus sampling. Defaults to 1.0.

The random seed for reproducibility. Defaults to 42.

The maximum number of tokens allowed in the completion.

Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.

Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.

The voice kind. Always azure-standard.

The Azure neural voice name.

The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.

Configures default conversation behavior and optional run-wide persistence of generated conversations.

Hard limit on turns in each conversation. When omitted, the service defaults to 20.

Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.

Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.

Configures effects applied to simulated conversation audio.

Effects to apply to simulated audio.

Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.

Configures conversation behavior for the simulated user.

Configuration for simulated user interruption behavior.

Configures the default interruption behavior for the simulated user.

The interruption type, always default.

Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.

Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.

Maps JSONL dataset attributes to user conversation simulation inputs when their names differ from the defaults. The default attribute names are test_case_id, test_case_category, test_case_description, and simulation_configuration, matching UserConversationSimulationTestCase. Omit when the dataset uses these names or when using inline_user_conversation_simulation.

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

An object representing an error response from the Eval API.

The error code.

The error message.

The name of the person who created the evaluation run.

Set of immutable 16 key-value pairs that can be attached to an object for storing additional information. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

The level at which evaluation is performed.

Datasets produced by the evaluation run. Omitted when no datasets were generated.

Latency measurements reported for an evaluation run.

Run-wide end-to-end latency percentiles for the evaluation target.

Median target latency in milliseconds.

95th percentile target latency in milliseconds.

Number of evaluation rows that contributed a usable latency measurement.

Estimated costs reported for an evaluation run.

Run-wide estimated inference cost for the evaluation target. The estimate is derived from reported token counts and list prices. It excludes evaluator and runtime usage and does not account for negotiated pricing, commitments, or discounts.

Estimated target inference cost in the specified currency.

ISO 4217 currency code for the estimate.

Indicates whether all target models attributed to the evaluation run were priced.

Identifier of the price-list snapshot used for the estimate.

Estimated cost and token usage for each priced target model.

Target models for which no reliable price was available.

The public explanation for this block.

The public continuation instruction.

The public explanation for this block.

The public continuation instruction.