Evaluations - Create Agent Evaluation

Service:

AI Foundry

API Version:

2025-05-15-preview

Creates an agent evaluation run.

POST {endpoint}/evaluations/runs:runAgent?api-version=2025-05-15-preview

URI Parameters

Name	In	Required	Type	Description
endpoint	path	True	string (uri)	Project endpoint. In the form "https://<your-ai-services-account-name>.services.ai.azure.com/api/projects/_project" if your Foundry Hub has only one Project, or to use the default Project in your Hub. Or in the form "https://<your-ai-services-account-name>.services.ai.azure.com/api/projects/<your-project-name>" if you want to explicitly specify the Foundry Project name.
api-version	query	True	string minLength: 1	The API version to use for this operation.

Request Body

Name	Required	Type	Description
appInsightsConnectionString	True	string	Pass the AppInsights connection string to the agent evaluation for the evaluation results and the errors logs.
evaluators	True	<string,  EvaluatorConfiguration>	Evaluators to be used for the evaluation.
runId	True	string	Identifier of the agent run.
redactionConfiguration		AgentEvaluationRedactionConfiguration	Redaction configuration for the evaluation.
samplingConfiguration		AgentEvaluationSamplingConfiguration	Sampling configuration for the evaluation.
threadId		string	Identifier of the agent thread. This field is mandatory currently, but it will be optional in the future.

Responses

Name	Type	Description
201 Created	AgentEvaluation	The request has succeeded and a new resource has been created as a result.
Other Status Codes	Azure.Core.Foundations.ErrorResponse	An unexpected error response. Headers x-ms-error-code: string

Security

OAuth2Auth

Type: oauth2
Flow: implicit
Authorization URL: https://login.microsoftonline.com/common/oauth2/v2.0/authorize

Scopes

Name	Description
https://ai.azure.com/.default

Examples

Evaluations_CreateAgentEvaluation_MaximumSet

Evaluations_CreateAgentEvaluation_MinimumSet

Evaluations_CreateAgentEvaluation_MaximumSet

Sample request

HTTP

POST {endpoint}/evaluations/runs:runAgent?api-version=2025-05-15-preview

{
  "runId": "ecfcffqsrdhpecocrufffiqz",
  "threadId": "cxjjsyhkeezgnaqqyerrdrbbth",
  "evaluators": {
    "key2653": {
      "id": "gujwtvhptykq",
      "initParams": {},
      "dataMapping": {
        "key7400": "ijkjfvoswni"
      }
    }
  },
  "samplingConfiguration": {
    "name": "tj",
    "samplingPercent": 7,
    "maxRequestRate": 8
  },
  "redactionConfiguration": {
    "redactScoreProperties": true
  },
  "appInsightsConnectionString": "dvcnrcwar"
}

Sample response

Status code:

201

{
  "id": "gji",
  "status": "ozywrhiasll",
  "error": "stcjbhxgmqvjqwzmbvaa",
  "result": [
    {
      "evaluator": "upnt",
      "evaluatorId": "upnt",
      "score": 13,
      "status": "eoxw",
      "reason": "kfrjetziuketgioobeydlugonzfxo",
      "version": "lfqwtzuktnbmcxxwkghmlpdtuu",
      "threadId": "sldwqklyuxeayfmxpbl",
      "runId": "lauehlf",
      "error": "lrutmshl",
      "additionalDetails": {
        "key3590": "vhhneubavtiklsjcdexij"
      }
    }
  ]
}

Evaluations_CreateAgentEvaluation_MinimumSet

Sample request

HTTP

POST {endpoint}/evaluations/runs:runAgent?api-version=2025-05-15-preview

{
  "runId": "ecfcffqsrdhpecocrufffiqz",
  "evaluators": {},
  "appInsightsConnectionString": "dvcnrcwar"
}

Sample response

Status code:

201

{
  "id": "gji",
  "status": "ozywrhiasll"
}

Definitions

Name	Description
AgentEvaluation	Evaluation response for agent evaluation run.
AgentEvaluationRedactionConfiguration	The redaction configuration will allow the user to control what is redacted.
AgentEvaluationRequest	Evaluation request for agent run.
AgentEvaluationResult	Result for the agent evaluation evaluator run.
AgentEvaluationSamplingConfiguration	Definition for sampling strategy.
Azure.Core.Foundations.Error	The error object.
Azure.Core.Foundations.ErrorResponse	A response containing error details.
Azure.Core.Foundations.InnerError	An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/microsoft/api-guidelines/blob/vNext/azure/Guidelines.md#handling-errors.
EvaluatorConfiguration	Evaluator Configuration

AgentEvaluation

Object

Evaluation response for agent evaluation run.

Name	Type	Description
error	string	The reason of the request failure for the long running process, if applicable.
id	string	Identifier of the agent evaluation run.
result	AgentEvaluationResult[]	The agent evaluation result.
status	string	Status of the agent evaluation. Options: Running, Completed, Failed.

AgentEvaluationRedactionConfiguration

Object

The redaction configuration will allow the user to control what is redacted.

Name	Type	Description
redactScoreProperties	boolean	Redact score properties. If not specified, the default is to redact in production.

AgentEvaluationRequest

Object

Evaluation request for agent run.

Name	Type	Description
appInsightsConnectionString	string	Pass the AppInsights connection string to the agent evaluation for the evaluation results and the errors logs.
evaluators	<string,  EvaluatorConfiguration>	Evaluators to be used for the evaluation.
redactionConfiguration	AgentEvaluationRedactionConfiguration	Redaction configuration for the evaluation.
runId	string	Identifier of the agent run.
samplingConfiguration	AgentEvaluationSamplingConfiguration	Sampling configuration for the evaluation.
threadId	string	Identifier of the agent thread. This field is mandatory currently, but it will be optional in the future.

AgentEvaluationResult

Object

Result for the agent evaluation evaluator run.

Name	Type	Description
additionalDetails	object	Additional properties relevant to the evaluator. These will differ between evaluators.
error	string	A string explaining why there was an error, if applicable.
evaluator	string	Evaluator's name. This is the name of the evaluator that was used to evaluate the agent's completion.
evaluatorId	string	Identifier of the evaluator.
reason	string	Reasoning for the evaluation result.
runId	string	The unique identifier of the run.
score	number (float)	Score of the given evaluator. No restriction on range.
status	string	Status of the evaluator result. Options: Running, Completed, Failed, NotApplicable.
threadId	string	The unique identifier of the thread.
version	string	Version of the evaluator that was used to evaluate the agent's completion.

AgentEvaluationSamplingConfiguration

Object

Definition for sampling strategy.

Name	Type	Description
maxRequestRate	number (float)	Maximum request rate per hour (0 to 1000).
name	string	Name of the sampling strategy.
samplingPercent	number (float)	Percentage of sampling per hour (0-100).

Azure.Core.Foundations.Error

Object

The error object.

Name	Type	Description
code	string	One of a server-defined set of error codes.
details	Azure.Core.Foundations.Error[]	An array of details about specific errors that led to this reported error.
innererror	Azure.Core.Foundations.InnerError	An object containing more specific information than the current object about the error.
message	string	A human-readable representation of the error.
target	string	The target of the error.

Azure.Core.Foundations.ErrorResponse

Object

A response containing error details.

Name	Type	Description
error	Azure.Core.Foundations.Error	The error object.

Azure.Core.Foundations.InnerError

Object

An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/microsoft/api-guidelines/blob/vNext/azure/Guidelines.md#handling-errors.

Name	Type	Description
code	string	One of a server-defined set of error codes.
innererror	Azure.Core.Foundations.InnerError	Inner error.

EvaluatorConfiguration

Object

Evaluator Configuration

Name	Type	Description
dataMapping	object	Data parameters of the evaluator.
id	string	Identifier of the evaluator.
initParams		Initialization parameters of the evaluator.