Evals
Create an evaluation
Creates an evaluation configuration that defines the data source schema and testing criteria used to measure model or agent performance.
Creating an evaluation does not execute it. To process data with the evaluation, create an evaluation run and provide a compatible data source.
Isi Permintaan
name
string
The name of the evaluation.
metadata
OpenAI.Metadata | null
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
data_source_config
OpenAI.CreateEvalCustomDataSourceConfig | OpenAI.CreateEvalLogsDataSourceConfig | OpenAI.CreateEvalStoredCompletionsDataSourceConfig | AzureAIDataSourceConfig | AzureAIBenchmarkDataSourceConfig
diperlukan
Defines the schema and kind of data that evaluation runs must provide. This property configures the evaluation; it does not contain the run's data. Supply the actual dataset, inline items, stored responses, traces, or target configuration in data_source when creating an evaluation run.
type
enum
diperlukan
The type of data source. Always custom.
item_schema
object
diperlukan
The json schema for each row in the data source.
include_sample_schema
boolean
Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
type
enum
diperlukan
The type of data source. Always logs.
metadata
object
Metadata filters for the logs data source.
type
enum
diperlukan
The type of data source. Always stored_completions.
metadata
object
Metadata filters for the stored completions data source.
benchmark_preview
AzureAIBenchmarkDataSourceConfig
Data source configuration for benchmark evaluations.
scenario
enum
diperlukan
Data schema scenario, always benchmark for benchmark evaluations.
benchmark_name
string | enum
diperlukan
The set of available benchmark specifications.
enum
benchmark_version
string
The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.
grader_model
string
Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.
scenario
enum
diperlukan
Data schema scenario, always benchmark for benchmark evaluations.
benchmark_name
string | enum
diperlukan
The set of available benchmark specifications.
enum
benchmark_version
string
The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.
grader_model
string
Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.
testing_criteria
(OpenAI.EvalGraderLabelModel | OpenAI.EvalGraderStringCheck | OpenAI.EvalGraderTextSimilarity | OpenAI.EvalGraderPython | OpenAI.EvalGraderScoreModel | TestingCriterionAzureAIEvaluator | EvalGraderInspectAI)[]
diperlukan
The graders and Azure AI evaluators to run against each evaluation item. Each testing criterion has a caller-defined name and a type-specific configuration. At least one testing criterion is required.
properties
object
Set of immutable 16 key-value pairs that can be attached to an object for storing additional information. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
Respons
The request has succeeded.
Isi
object
enum
diperlukan
The object type.
id
string
diperlukan
Unique identifier for the evaluation.
name
string
diperlukan
The name of the evaluation.
data_source_config
OpenAI.CreateEvalCustomDataSourceConfig | OpenAI.CreateEvalLogsDataSourceConfig | OpenAI.CreateEvalStoredCompletionsDataSourceConfig | AzureAIDataSourceConfig | AzureAIBenchmarkDataSourceConfig
diperlukan
Configuration of data sources used in runs of the evaluation.
type
enum
diperlukan
The type of data source. Always custom.
item_schema
object
diperlukan
The json schema for each row in the data source.
include_sample_schema
boolean
Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
type
enum
diperlukan
The type of data source. Always logs.
metadata
object
Metadata filters for the logs data source.
type
enum
diperlukan
The type of data source. Always stored_completions.
metadata
object
Metadata filters for the stored completions data source.
benchmark_preview
AzureAIBenchmarkDataSourceConfig
Data source configuration for benchmark evaluations.
scenario
enum
diperlukan
Data schema scenario, always benchmark for benchmark evaluations.
benchmark_name
string | enum
diperlukan
The set of available benchmark specifications.
enum
benchmark_version
string
The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.
grader_model
string
Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.
scenario
enum
diperlukan
Data schema scenario, always benchmark for benchmark evaluations.
benchmark_name
string | enum
diperlukan
The set of available benchmark specifications.
enum
benchmark_version
string
The version of the benchmark specification (e.g., '0.1'). Latest version if not specified.
grader_model
string
Optional grader model for benchmarks that use model graders, Required when the benchmark's testing_criteria uses a model grader type.
testing_criteria
(OpenAI.EvalGraderLabelModel | OpenAI.EvalGraderStringCheck | OpenAI.EvalGraderTextSimilarity | OpenAI.EvalGraderPython | OpenAI.EvalGraderScoreModel | TestingCriterionAzureAIEvaluator | EvalGraderInspectAI)[]
diperlukan
A list of testing criteria.
created_at
integer
diperlukan
The Unix timestamp (in seconds) for when the eval was created.
metadata
OpenAI.Metadata | null
diperlukan
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
created_by
string
The name of the person who created the evaluation.
properties
object
Set of immutable 16 key-value pairs that can be attached to an object for storing additional information. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
Minta
curl https://api.openai.com/v1/evals \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Sentiment",
"data_source_config": {
"type": "stored_completions",
"metadata": {
"usecase": "chatbot"
}
},
"testing_criteria": [
{
"type": "label_model",
"model": "o3-mini",
"input": [
{
"role": "developer",
"content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"
},
{
"role": "user",
"content": "Statement: {{item.input}}"
}
],
"passing_labels": [
"positive"
],
"labels": [
"positive",
"neutral",
"negative"
],
"name": "Example label grader"
}
]
}'import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted
)
eval = client.evals.create(
data_source_config={
"item_schema": {
"foo": "bar"
},
"type": "custom",
},
testing_criteria=[{
"input": [{
"content": "content",
"role": "role",
}],
"labels": ["string"],
"model": "model",
"name": "name",
"passing_labels": ["string"],
"type": "label_model",
}],
)
print(eval.id)import OpenAI from "openai";
const openai = new OpenAI();
const evalObj = await openai.evals.create({
name: "Sentiment",
data_source_config: {
type: "stored_completions",
metadata: { usecase: "chatbot" }
},
testing_criteria: [
{
type: "label_model",
model: "o3-mini",
input: [
{ role: "developer", content: "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'" },
{ role: "user", content: "Statement: {{item.input}}" }
],
passing_labels: ["positive"],
labels: ["positive", "neutral", "negative"],
name: "Example label grader"
}
]
});
console.log(evalObj);import OpenAI from 'openai';
const client = new OpenAI({
apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted
});
const _eval = await client.evals.create({
data_source_config: {
item_schema: { foo: 'bar' },
type: 'custom',
},
testing_criteria: [
{
input: [{ content: 'content', role: 'role' }],
labels: ['string'],
model: 'model',
name: 'name',
passing_labels: ['string'],
type: 'label_model',
},
],
});
console.log(_eval.id);package com.openai.example;
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.evals.EvalCreateParams;
import com.openai.models.evals.EvalCreateResponse;
public final class Main {
private Main() {}
public static void main(String[] args) {
OpenAIClient client = OpenAIOkHttpClient.fromEnv();
EvalCreateParams params = EvalCreateParams.builder()
.customDataSourceConfig(EvalCreateParams.DataSourceConfig.Custom.ItemSchema.builder()
.putAdditionalProperty("foo", JsonValue.from("bar"))
.build())
.addTestingCriterion(EvalCreateParams.TestingCriterion.LabelModel.builder()
.addInput(EvalCreateParams.TestingCriterion.LabelModel.Input.SimpleInputMessage.builder()
.content("content")
.role("role")
.build())
.addLabel("string")
.model("model")
.name("name")
.addPassingLabel("string")
.build())
.build();
EvalCreateResponse eval = client.evals().create(params);
}
}require "openai"
openai = OpenAI::Client.new(api_key: "My API Key")
eval_ = openai.evals.create(
data_source_config: {item_schema: {foo: "bar"}, type: :custom},
testing_criteria: [
{
input: [{content: "content", role: "role"}],
labels: ["string"],
model: "model",
name: "name",
passing_labels: ["string"],
type: :label_model
}
]
)
puts(eval_)Tanggapan
{
"object": "eval",
"id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
"data_source_config": {
"type": "stored_completions",
"metadata": {
"usecase": "chatbot"
},
"schema": {
"type": "object",
"properties": {
"item": {
"type": "object"
},
"sample": {
"type": "object"
}
},
"required": [
"item",
"sample"
]
},
"testing_criteria": [
{
"name": "Example label grader",
"type": "label_model",
"model": "o3-mini",
"input": [
{
"type": "message",
"role": "developer",
"content": {
"type": "input_text",
"text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
}
},
{
"type": "message",
"role": "user",
"content": {
"type": "input_text",
"text": "Statement: {{item.input}}"
}
}
],
"passing_labels": [
"positive"
],
"labels": [
"positive",
"neutral",
"negative"
]
}
],
"name": "Sentiment",
"created_at": 1740110490,
"metadata": {
"description": "An eval for sentiment analysis"
}
}