Skip to main content

This browser is no longer supported.

Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.

Download Microsoft Edge More info about Internet Explorer and Microsoft Edge
microsoft-foundry.rest.security.o-auth2-auth

Fine-tuning

Run a grader

POST /openai/v1/fine_tuning/alpha/graders/run

Runs a grader.

Request Body

The request body.

Content-Type: application/json
grader OpenAI.GraderStringCheck | OpenAI.GraderTextSimilarity | OpenAI.GraderPython | OpenAI.GraderScoreModel | OpenAI.GraderMulti required

The grader used for the fine-tuning job.

One of the following:
type enum required

The object type, which is always string_check.

string_check
name string required

The name of the grader.

input string required

The input text. This may include template strings.

reference string required

The reference text. This may include template strings.

operation enum required

The string check operation to perform. One of eq, ne, like, or ilike.

eqnelikeilike
type enum required

The type of grader.

text_similarity
name string required

The name of the grader.

input string required

The text being graded.

reference string required

The text being graded against.

evaluation_metric enum required

The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.

cosinefuzzy_matchbleugleumeteorrouge_1rouge_2rouge_3rouge_4rouge_5rouge_l
type enum required

The object type, which is always python.

python
name string required

The name of the grader.

source string required

The source code of the python script.

image_tag string

The image tag to use for the python script.

type enum required

The object type, which is always score_model.

score_model
name string required

The name of the grader.

model string required

The model to use for the evaluation.

sampling_params OpenAI.EvalGraderScoreModelSamplingParams
seed integer | null
top_p number | null
temperature number | null
max_completions_tokens integer | null
reasoning_effort string | enum | null

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

Any of the following:
string
enum
noneminimallowmediumhighxhighmax
input OpenAI.EvalItem[] required

The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

range OpenAI.numeric[]

The service requires two numbers for the score range. Defaults to [0, 1].

type enum required

The object type, which is always multi.

multi
name string required

The name of the grader.

graders object required
[key: string] OpenAI.GraderStringCheck | OpenAI.GraderTextSimilarity | OpenAI.GraderPython | OpenAI.GraderScoreModel | OpenAI.GraderLabelModel
Any of the following:
type enum required

The object type, which is always string_check.

string_check
name string required

The name of the grader.

input string required

The input text. This may include template strings.

reference string required

The reference text. This may include template strings.

operation enum required

The string check operation to perform. One of eq, ne, like, or ilike.

eqnelikeilike
type enum required

The type of grader.

text_similarity
name string required

The name of the grader.

input string required

The text being graded.

reference string required

The text being graded against.

evaluation_metric enum required

The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.

cosinefuzzy_matchbleugleumeteorrouge_1rouge_2rouge_3rouge_4rouge_5rouge_l
type enum required

The object type, which is always python.

python
name string required

The name of the grader.

source string required

The source code of the python script.

image_tag string

The image tag to use for the python script.

type enum required

The object type, which is always score_model.

score_model
name string required

The name of the grader.

model string required

The model to use for the evaluation.

sampling_params OpenAI.EvalGraderScoreModelSamplingParams
seed integer | null
top_p number | null
temperature number | null
max_completions_tokens integer | null
reasoning_effort string | enum | null

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

Any of the following:
string
enum
noneminimallowmediumhighxhighmax
input OpenAI.EvalItem[] required

The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

range OpenAI.numeric[]

The service requires two numbers for the score range. Defaults to [0, 1].

type enum required

The object type, which is always label_model.

label_model
name string required

The name of the grader.

model string required

The model to use for the evaluation. Must support structured outputs.

input OpenAI.EvalItem[] required
labels string[] required

The labels to assign to each item in the evaluation.

passing_labels string[] required

The labels that indicate a passing result. Must be a subset of labels.

calculate_output string required

A formula to calculate the output based on grader results.

item OpenAI.RunGraderRequestItem
model_sample string required

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.

Responses

The request has succeeded.

Body

Content-Type: application/json
reward number required
metadata OpenAI.RunGraderResponseMetadata required
name string required
type string required
errors OpenAI.RunGraderResponseMetadataErrors required
formula_parse_error boolean required
sample_parse_error boolean required
truncated_observation_error boolean required
unresponsive_reward_error boolean required
invalid_variable_error boolean required
other_error boolean required
python_grader_server_error boolean required
python_grader_server_error_type string | null required
python_grader_runtime_error boolean required
python_grader_runtime_error_details string | null required
model_grader_server_error boolean required
model_grader_refusal_error boolean required
model_grader_parse_error boolean required
model_grader_server_error_details string | null required
execution_time number required
scores object required
token_usage OpenAI.RunGraderResponseMetadataTokenUsage | null required
prompt_tokens integer required
total_tokens integer required
completion_tokens integer | null
cached_tokens integer | null
sampled_model_name string | null required
sub_rewards object required
model_grader_token_usage_per_model object required

Client error

Body

Content-Type: application/json
error OpenAI.Error required
code string | null required
message string required
param string | null
type string
misalignment OpenAI.MisalignmentErrorDetailsResource
review_target string | null
error_type string | enum
Any of the following:
string
enum
potentially_unintended_data_transferpotentially_unintended_data_accesspotentially_unintended_destructive_activityother
detailed_explanation string

The public explanation for this block.

steer OpenAI._MisalignmentSteer
message string required

The public continuation instruction.

details OpenAI.Error[]
additionalInfo object
debugInfo object

Server error

Body

Content-Type: application/json
error OpenAI.Error required
code string | null required
message string required
param string | null
type string
misalignment OpenAI.MisalignmentErrorDetailsResource
review_target string | null
error_type string | enum
Any of the following:
string
enum
potentially_unintended_data_transferpotentially_unintended_data_accesspotentially_unintended_destructive_activityother
detailed_explanation string

The public explanation for this block.

steer OpenAI._MisalignmentSteer
message string required

The public continuation instruction.

details OpenAI.Error[]
additionalInfo object
debugInfo object

Request

curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "grader": {
      "type": "score_model",
      "name": "Example score model grader",
      "input": [
        {
          "role": "user",
          "content": [
            {
              "type": "input_text",
              "text": "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score.

Reference answer: {{item.reference_answer}}

Model answer: {{sample.output_text}}"
            }
          ]
        }
      ],
      "model": "gpt-5-mini",
      "sampling_params": {
        "temperature": 1,
        "top_p": 1,
        "seed": 42
      }
    },
    "item": {
      "reference_answer": "fuzzy wuzzy was a bear"
    },
    "model_sample": "fuzzy wuzzy was a bear"
  }'
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # This is the default and can be omitted
)
response = client.fine_tuning.alpha.graders.run(
    grader={
        "input": "input",
        "name": "name",
        "operation": "eq",
        "reference": "reference",
        "type": "string_check",
    },
    model_sample="model_sample",
)
print(response.metadata)
import OpenAI from "openai";

const openai = new OpenAI();

const result = await openai.fineTuning.alpha.graders.run({
  grader: {
    type: "score_model",
    name: "Example score model grader",
    input: [
      {
        role: "user",
        content: [
          {
            type: "input_text",
            text: "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score.

Reference answer: {{item.reference_answer}}

Model answer: {{sample.output_text}}",
          },
        ],
      },
    ],
    model: "gpt-5-mini",
    sampling_params: { temperature: 1, top_p: 1, seed: 42 },
  },
  item: { reference_answer: "fuzzy wuzzy was a bear" },
  model_sample: "fuzzy wuzzy was a bear",
});
console.log(result);
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted
});

const response = await client.fineTuning.alpha.graders.run({
  grader: {
    input: 'input',
    name: 'name',
    operation: 'eq',
    reference: 'reference',
    type: 'string_check',
  },
  model_sample: 'model_sample',
});

console.log(response.metadata);
package main

import (
    "context"
    "fmt"

    "github.com/openai/openai-go"
    "github.com/openai/openai-go/option"
)

func main() {
    client := openai.NewClient(
        option.WithAPIKey("My API Key"),
    )
    response, err := client.FineTuning.Alpha.Graders.Run(context.TODO(), openai.FineTuningAlphaGraderRunParams{
        Grader: openai.FineTuningAlphaGraderRunParamsGraderUnion{
            OfStringCheck: &openai.StringCheckGraderParam{
                Input:     "input",
                Name:      "name",
                Operation: openai.StringCheckGraderOperationEq,
                Reference: "reference",
            },
        },
        ModelSample: "model_sample",
    })
    if err != nil {
        panic(err.Error())
    }
    fmt.Printf("%+v
", response.Metadata)
}
package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.finetuning.alpha.graders.GraderRunParams;
import com.openai.models.finetuning.alpha.graders.GraderRunResponse;
import com.openai.models.graders.gradermodels.StringCheckGrader;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        GraderRunParams params = GraderRunParams.builder()
            .grader(StringCheckGrader.builder()
                .input("input")
                .name("name")
                .operation(StringCheckGrader.Operation.EQ)
                .reference("reference")
                .build())
            .modelSample("model_sample")
            .build();
        GraderRunResponse response = client.fineTuning().alpha().graders().run(params);
    }
}
require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.run(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},
  model_sample: "model_sample"
)

puts(response)

Response

{
  "reward": 1.0,
  "metadata": {
    "name": "Example score model grader",
    "type": "score_model",
    "errors": {
      "formula_parse_error": false,
      "sample_parse_error": false,
      "truncated_observation_error": false,
      "unresponsive_reward_error": false,
      "invalid_variable_error": false,
      "other_error": false,
      "python_grader_server_error": false,
      "python_grader_server_error_type": null,
      "python_grader_runtime_error": false,
      "python_grader_runtime_error_details": null,
      "model_grader_server_error": false,
      "model_grader_refusal_error": false,
      "model_grader_parse_error": false,
      "model_grader_server_error_details": null
    },
    "execution_time": 4.365238428115845,
    "scores": {},
    "token_usage": {
      "prompt_tokens": 190,
      "total_tokens": 324,
      "completion_tokens": 134,
      "cached_tokens": 0
    },
    "sampled_model_name": "gpt-4o-2024-08-06"
  },
  "sub_rewards": {},
  "model_grader_token_usage_per_model": {
    "gpt-4o-2024-08-06": {
      "prompt_tokens": 190,
      "total_tokens": 324,
      "completion_tokens": 134,
      "cached_tokens": 0
    }
  }
}

Request

curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "grader": {
      "type": "score_model",
      "name": "Image caption grader",
      "input": [
        {
          "role": "user",
          "content": [
            {
              "type": "input_text",
              "text": "Score how well the provided caption matches the image on a 0-1 scale. Only return the score.

Caption: {{sample.output_text}}"
            },
            {
              "type": "input_image",
              "image_url": "https://example.com/dog-catching-ball.png",
              "file_id": null,
              "detail": "high"
            }
          ]
        }
      ],
      "model": "gpt-5-mini",
      "sampling_params": {
        "temperature": 0.2
      }
    },
    "item": {
      "expected_caption": "A golden retriever jumps to catch a tennis ball"
    },
    "model_sample": "A dog leaps to grab a tennis ball mid-air"
  }'
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted
});

const response = await client.fineTuning.alpha.graders.run({
  grader: {
    input: 'input',
    name: 'name',
    operation: 'eq',
    reference: 'reference',
    type: 'string_check',
  },
  model_sample: 'model_sample',
});

console.log(response.metadata);
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # This is the default and can be omitted
)
response = client.fine_tuning.alpha.graders.run(
    grader={
        "input": "input",
        "name": "name",
        "operation": "eq",
        "reference": "reference",
        "type": "string_check",
    },
    model_sample="model_sample",
)
print(response.metadata)
package main

import (
    "context"
    "fmt"

    "github.com/openai/openai-go"
    "github.com/openai/openai-go/option"
)

func main() {
    client := openai.NewClient(
        option.WithAPIKey("My API Key"),
    )
    response, err := client.FineTuning.Alpha.Graders.Run(context.TODO(), openai.FineTuningAlphaGraderRunParams{
        Grader: openai.FineTuningAlphaGraderRunParamsGraderUnion{
            OfStringCheck: &openai.StringCheckGraderParam{
                Input:     "input",
                Name:      "name",
                Operation: openai.StringCheckGraderOperationEq,
                Reference: "reference",
            },
        },
        ModelSample: "model_sample",
    })
    if err != nil {
        panic(err.Error())
    }
    fmt.Printf("%+v
", response.Metadata)
}
package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.finetuning.alpha.graders.GraderRunParams;
import com.openai.models.finetuning.alpha.graders.GraderRunResponse;
import com.openai.models.graders.gradermodels.StringCheckGrader;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        GraderRunParams params = GraderRunParams.builder()
            .grader(StringCheckGrader.builder()
                .input("input")
                .name("name")
                .operation(StringCheckGrader.Operation.EQ)
                .reference("reference")
                .build())
            .modelSample("model_sample")
            .build();
        GraderRunResponse response = client.fineTuning().alpha().graders().run(params);
    }
}
require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.run(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},
  model_sample: "model_sample"
)

puts(response)

Request

curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "grader": {
      "type": "score_model",
      "name": "Audio clarity grader",
      "input": [
        {
          "role": "user",
          "content": [
            {
              "type": "input_text",
              "text": "Listen to the clip and return a confidence score from 0 to 1 that the speaker said: {{item.target_phrase}}"
            },
            {
              "type": "input_audio",
              "input_audio": {
                "data": "{{item.audio_clip_b64}}",
                "format": "mp3"
              }
            }
          ]
        }
      ],
      "model": "gpt-audio",
      "sampling_params": {
        "temperature": 0.2,
        "top_p": 1,
        "seed": 123
      }
    },
    "item": {
      "target_phrase": "Please deliver the package on Tuesday",
      "audio_clip_b64": "<base64-encoded mp3>"
    },
    "model_sample": "Please deliver the package on Tuesday"
  }'
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted
});

const response = await client.fineTuning.alpha.graders.run({
  grader: {
    input: 'input',
    name: 'name',
    operation: 'eq',
    reference: 'reference',
    type: 'string_check',
  },
  model_sample: 'model_sample',
});

console.log(response.metadata);
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # This is the default and can be omitted
)
response = client.fine_tuning.alpha.graders.run(
    grader={
        "input": "input",
        "name": "name",
        "operation": "eq",
        "reference": "reference",
        "type": "string_check",
    },
    model_sample="model_sample",
)
print(response.metadata)
package main

import (
    "context"
    "fmt"

    "github.com/openai/openai-go"
    "github.com/openai/openai-go/option"
)

func main() {
    client := openai.NewClient(
        option.WithAPIKey("My API Key"),
    )
    response, err := client.FineTuning.Alpha.Graders.Run(context.TODO(), openai.FineTuningAlphaGraderRunParams{
        Grader: openai.FineTuningAlphaGraderRunParamsGraderUnion{
            OfStringCheck: &openai.StringCheckGraderParam{
                Input:     "input",
                Name:      "name",
                Operation: openai.StringCheckGraderOperationEq,
                Reference: "reference",
            },
        },
        ModelSample: "model_sample",
    })
    if err != nil {
        panic(err.Error())
    }
    fmt.Printf("%+v
", response.Metadata)
}
package com.openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.finetuning.alpha.graders.GraderRunParams;
import com.openai.models.finetuning.alpha.graders.GraderRunResponse;
import com.openai.models.graders.gradermodels.StringCheckGrader;

public final class Main {
    private Main() {}

    public static void main(String[] args) {
        OpenAIClient client = OpenAIOkHttpClient.fromEnv();

        GraderRunParams params = GraderRunParams.builder()
            .grader(StringCheckGrader.builder()
                .input("input")
                .name("name")
                .operation(StringCheckGrader.Operation.EQ)
                .reference("reference")
                .build())
            .modelSample("model_sample")
            .build();
        GraderRunResponse response = client.fineTuning().alpha().graders().run(params);
    }
}
require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.run(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},
  model_sample: "model_sample"
)

puts(response)
en-us
Your Privacy Choices
  • AI Disclaimer
  • Previous Versions
  • Blog
  • Contribute
  • Privacy
  • Consumer Health Privacy
  • Terms of Use
  • Trademarks
  • © Microsoft 2026