Hoppa till huvudinnehåll

Den här webbläsaren stöds inte längre.

Uppgradera till Microsoft Edge och dra nytta av de senaste funktionerna och säkerhetsuppdateringarna, samt teknisk support.

Ladda ned Microsoft Edge Mer information om Internet Explorer och Microsoft Edge
microsoft-foundry.rest.security.o-auth2-auth

Evals

Create an evaluation

POST /openai/v1/evals

Creates an evaluation configuration that defines the data source schema and testing criteria used to measure model or agent performance.

Creating an evaluation does not execute it. To process data with the evaluation, create an evaluation run and provide a compatible data source.

Begärandetext

Content-Type: application/json
name string

The name of the evaluation.

metadata OpenAI.Metadata | null

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

[key: string] string
data_source_config OpenAI.CreateEvalCustomDataSourceConfig | OpenAI.CreateEvalLogsDataSourceConfig | OpenAI.CreateEvalStoredCompletionsDataSourceConfig | AzureAIDataSourceConfig | AzureAIBenchmarkDataSourceConfig krävs

Defines the schema and kind of data that evaluation runs must provide. This property configures the evaluation; it does not contain the run's data. Supply the actual dataset, inline items, stored responses, traces, or target configuration in data_source when creating an evaluation run.

Något av följande:
type enum krävs

The type of data source. Always custom.

custom
item_schema object krävs

The json schema for each row in the data source.

include_sample_schema boolean

Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)

type enum krävs

The type of data source. Always logs.

logs
metadata object

Metadata filters for the logs data source.

type enum krävs

The type of data source. Always stored_completions.

stored_completions
metadata object

Metadata filters for the stored completions data source.

Något av följande:
benchmark_preview AzureAIBenchmarkDataSourceConfig

Data source configuration for benchmark evaluations.

scenario enum krävs

Data schema scenario, always benchmark for benchmark evaluations.

benchmark_preview
benchmark_name string | enum krävs

The set of available benchmark specifications.

Något av följande:
string
enum
builtin.gpqa_diamondbuiltin.bbehbuiltin.bigbenchhardbuiltin.frontiersciencebuiltin.ifevalbuiltin.musrbuiltin.truthful_qabuiltin.inspect_ai.aime2025builtin.inspect_ai.browse_compbuiltin.inspect_ai.chembenchbuiltin.inspect_ai.tau2_telecom
benchmark_version string

The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.

grader_model string

Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.

scenario enum krävs

Data schema scenario, always benchmark for benchmark evaluations.

benchmark_preview
benchmark_name string | enum krävs

The set of available benchmark specifications.

Något av följande:
string
enum
builtin.gpqa_diamondbuiltin.bbehbuiltin.bigbenchhardbuiltin.frontiersciencebuiltin.ifevalbuiltin.musrbuiltin.truthful_qabuiltin.inspect_ai.aime2025builtin.inspect_ai.browse_compbuiltin.inspect_ai.chembenchbuiltin.inspect_ai.tau2_telecom
benchmark_version string

The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.

grader_model string

Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.

testing_criteria (OpenAI.EvalGraderLabelModel | OpenAI.EvalGraderStringCheck | OpenAI.EvalGraderTextSimilarity | OpenAI.EvalGraderPython | OpenAI.EvalGraderScoreModel | TestingCriterionAzureAIEvaluator | EvalGraderInspectAI)[] krävs

The graders and Azure AI evaluators to run against each evaluation item. Each testing criterion has a caller-defined name and a type-specific configuration. At least one testing criterion is required.

properties object

Set of immutable 16 key-value pairs that can be attached to an object for storing additional information. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

[key: string] string

Svar

The request has succeeded.

Brödtext

Content-Type: application/json
object enum krävs

The object type.

eval
id string krävs

Unique identifier for the evaluation.

name string krävs

The name of the evaluation.

data_source_config OpenAI.CreateEvalCustomDataSourceConfig | OpenAI.CreateEvalLogsDataSourceConfig | OpenAI.CreateEvalStoredCompletionsDataSourceConfig | AzureAIDataSourceConfig | AzureAIBenchmarkDataSourceConfig krävs

Configuration of data sources used in runs of the evaluation.

Något av följande:
type enum krävs

The type of data source. Always custom.

custom
item_schema object krävs

The json schema for each row in the data source.

include_sample_schema boolean

Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)

type enum krävs

The type of data source. Always logs.

logs
metadata object

Metadata filters for the logs data source.

type enum krävs

The type of data source. Always stored_completions.

stored_completions
metadata object

Metadata filters for the stored completions data source.

Något av följande:
benchmark_preview AzureAIBenchmarkDataSourceConfig

Data source configuration for benchmark evaluations.

scenario enum krävs

Data schema scenario, always benchmark for benchmark evaluations.

benchmark_preview
benchmark_name string | enum krävs

The set of available benchmark specifications.

Något av följande:
string
enum
builtin.gpqa_diamondbuiltin.bbehbuiltin.bigbenchhardbuiltin.frontiersciencebuiltin.ifevalbuiltin.musrbuiltin.truthful_qabuiltin.inspect_ai.aime2025builtin.inspect_ai.browse_compbuiltin.inspect_ai.chembenchbuiltin.inspect_ai.tau2_telecom
benchmark_version string

The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.

grader_model string

Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.

scenario enum krävs

Data schema scenario, always benchmark for benchmark evaluations.

benchmark_preview
benchmark_name string | enum krävs

The set of available benchmark specifications.

Något av följande:
string
enum
builtin.gpqa_diamondbuiltin.bbehbuiltin.bigbenchhardbuiltin.frontiersciencebuiltin.ifevalbuiltin.musrbuiltin.truthful_qabuiltin.inspect_ai.aime2025builtin.inspect_ai.browse_compbuiltin.inspect_ai.chembenchbuiltin.inspect_ai.tau2_telecom
benchmark_version string

The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.

grader_model string

Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.

testing_criteria (OpenAI.EvalGraderLabelModel | OpenAI.EvalGraderStringCheck | OpenAI.EvalGraderTextSimilarity | OpenAI.EvalGraderPython | OpenAI.EvalGraderScoreModel | TestingCriterionAzureAIEvaluator | EvalGraderInspectAI)[] krävs

A list of testing criteria.

created_at integer krävs

The Unix timestamp (in seconds) for when the eval was created.

metadata OpenAI.Metadata | null krävs

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

[key: string] string
modified_at integer
created_by string

The name of the person who created the evaluation.

properties object

Set of immutable 16 key-value pairs that can be attached to an object for storing additional information. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

[key: string] string

Client error

Brödtext

Content-Type: application/json
error OpenAI.Error krävs
code string | null krävs
message string krävs
param string | null
type string
misalignment OpenAI.MisalignmentErrorDetailsResource
review_target string | null
error_type string | enum
Något av följande:
string
enum
potentially_unintended_data_transferpotentially_unintended_data_accesspotentially_unintended_destructive_activityother
detailed_explanation string

The public explanation for this block.

steer OpenAI._MisalignmentSteer
message string krävs

The public continuation instruction.

details OpenAI.Error[]
additionalInfo object
debugInfo object

Server error

Brödtext

Content-Type: application/json
error OpenAI.Error krävs
code string | null krävs
message string krävs
param string | null
type string
misalignment OpenAI.MisalignmentErrorDetailsResource
review_target string | null
error_type string | enum
Något av följande:
string
enum
potentially_unintended_data_transferpotentially_unintended_data_accesspotentially_unintended_destructive_activityother
detailed_explanation string

The public explanation for this block.

steer OpenAI._MisalignmentSteer
message string krävs

The public continuation instruction.

details OpenAI.Error[]
additionalInfo object
debugInfo object

Förfrågan

curl https://api.openai.com/v1/evals \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "Sentiment",
        "data_source_config": {
          "type": "stored_completions",
          "metadata": {
              "usecase": "chatbot"
          }
        },
        "testing_criteria": [
          {
            "type": "label_model",
            "model": "o3-mini",
            "input": [
              {
                "role": "developer",
                "content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"
              },
              {
                "role": "user",
                "content": "Statement: {{item.input}}"
              }
            ],
            "passing_labels": [
              "positive"
            ],
            "labels": [
              "positive",
              "neutral",
              "negative"
            ],
            "name": "Example label grader"
          }
        ]
      }'
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # This is the default and can be omitted
)
eval = client.evals.create(
    data_source_config={
        "item_schema": {
            "foo": "bar"
        },
        "type": "custom",
    },
    testing_criteria=[{
        "input": [{
            "content": "content",
            "role": "role",
        }],
        "labels": ["string"],
        "model": "model",
        "name": "name",
        "passing_labels": ["string"],
        "type": "label_model",
    }],
)
print(eval.id)
import OpenAI from "openai";

const openai = new OpenAI();

const evalObj = await openai.evals.create({
  name: "Sentiment",
  data_source_config: {
    type: "stored_completions",
    metadata: { usecase: "chatbot" }
  },
  testing_criteria: [
    {
      type: "label_model",
      model: "o3-mini",
      input: [
        { role: "developer", content: "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'" },
        { role: "user", content: "Statement: {{item.input}}" }
      ],
      passing_labels: ["positive"],
      labels: ["positive", "neutral", "negative"],
      name: "Example label grader"
    }
  ]
});
console.log(evalObj);
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted
});

const _eval = await client.evals.create({
  data_source_config: {
    item_schema: { foo: 'bar' },
    type: 'custom',
  },
  testing_criteria: [
    {
      input: [{ content: 'content', role: 'role' }],
      labels: ['string'],
      model: 'model',
      name: 'name',
      passing_labels: ['string'],
      type: 'label_model',
    },
  ],
});

console.log(_eval.id);
package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.evals.EvalCreateParams;
import com.openai.models.evals.EvalCreateResponse;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        EvalCreateParams params = EvalCreateParams.builder()
            .customDataSourceConfig(EvalCreateParams.DataSourceConfig.Custom.ItemSchema.builder()
                .putAdditionalProperty("foo", JsonValue.from("bar"))
                .build())
            .addTestingCriterion(EvalCreateParams.TestingCriterion.LabelModel.builder()
                .addInput(EvalCreateParams.TestingCriterion.LabelModel.Input.SimpleInputMessage.builder()
                    .content("content")
                    .role("role")
                    .build())
                .addLabel("string")
                .model("model")
                .name("name")
                .addPassingLabel("string")
                .build())
            .build();
        EvalCreateResponse eval = client.evals().create(params);
    }
}
require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

eval_ = openai.evals.create(
  data_source_config: {item_schema: {foo: "bar"}, type: :custom},
  testing_criteria: [
    {
      input: [{content: "content", role: "role"}],
      labels: ["string"],
      model: "model",
      name: "name",
      passing_labels: ["string"],
      type: :label_model
    }
  ]
)

puts(eval_)

Response

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}
sv-se
Dina sekretessalternativ
  • Ansvarsfriskrivning för AI
  • Tidigare versioner
  • Blogg
  • Bidra
  • Sekretess
  • Konsumenthälsans integritet
  • Användningsvillkor
  • Varumärken
  • © Microsoft 2026