EvaluatorGenerationJobs

Get an evaluator generation job

GET /evaluator_generation_jobs/{jobId}

Gets the details of an evaluator generation job by its ID.

Path parameters

jobId string required

The ID of the job.

Query parameters

api-version string required

The API version to use for this operation.

Responses

The request has succeeded.

Headers

Retry-After integer

Recommended number of seconds to wait before polling again.

Body

Content-Type: application/json

id string required

Server-assigned unique identifier.

result EvaluatorVersion

Evaluator Definition

display_name string

Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.

metadata object

Metadata about the evaluator

[key: string] string

evaluator_type string | enum required

The type of the evaluator

Any of the following:

string

enum

builtincustom

categories EvaluatorCategory[] required

The categories of the evaluator

supported_evaluation_levels EvaluationLevel[]

Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).

definition code | prompt | rubric | endpoint required

Base evaluator configuration with discriminator

One of the following:

code CodeBasedEvaluatorDefinition

Code-based evaluator definition using python code

type enum required

code

code_text string

Inline code text for the evaluator

entry_point string

The entry point Python file name for the uploaded evaluator code (e.g. 'answer_length_evaluator.py')

image_tag string

The container image tag to use for evaluator code execution

blob_uri string

The blob URI for the evaluator storage

prompt PromptBasedEvaluatorDefinition

Prompt-based evaluator

type enum required

prompt

prompt_text string required

The prompt text used for evaluation

rubric RubricBasedEvaluatorDefinition

Rubric-based evaluator definition — stores dimensions produced by the generate API. Used for both quality and safety evaluators.

type enum required

rubric

dimensions Dimension[] required

The set of dimensions — the scoring blueprint used by the LLM judge. Quality evaluators include a non-editable residual dimension with id 'general_quality' (always_applicable: true); safety evaluators include 'general_policy_compliance'. Both use the same Dimension structure.

pass_threshold number

Pass/fail threshold for the aggregate rubric score, on the same normalized 0.0-1.0 scale as the emitted score. When the runtime weighted average meets or exceeds this value, the result is pass. Defaults to 0.5 (equivalent to a raw 1-5 weighted average of 3.0). The 'any dimension scored 1 → fail' rule still applies regardless of this threshold.

endpoint EndpointBasedEvaluatorDefinition

Endpoint-based evaluator definition. The customer owns and hosts an HTTP endpoint that implements the evaluation contract. The evaluator references a Project Connection by name; the connection stores the endpoint URL and credentials (API Key or Entra ID). At execution time, the service resolves the connection to obtain the endpoint URL and authentication details, then calls the endpoint for each evaluation row.

type enum required

endpoint

connection_name string required

Name of the Project Connection that stores the endpoint URL and credentials. The connection must exist on the project and have a non-empty target URL. Supported auth types: ApiKey (sends api-key header) and AAD/Entra ID (acquires a bearer token via the project's Managed Identity).

generation_artifacts EvaluatorGenerationArtifacts

Service-managed provenance artifacts produced by an evaluator generation job. Present only on EvaluatorVersion resources created via the generation pipeline. The combined-JSONL Foundry Dataset is read-only and resolves to a versioned dataset in a service-reserved namespace.

dataset DatasetReference required

Reference to a versioned Foundry Dataset.

name string required

Dataset name.

version string required

Dataset version.

kinds string[] required

The kinds of rows present in dataset. Always contains "spec" (the generated evaluation specification, a Markdown document describing what the evaluator measures). May additionally contain "tools" (when the generation pipeline produced or inferred OpenAI tool schemas) and/or "context" (when supplementary materials such as file uploads or trace samples were used during generation).

generation_job_id string

Read-only provenance link back to the EvaluatorGenerationJob that produced this version. Present only on evaluator versions created via the generation pipeline; absent for manually-created versions and unaffected by subsequent PATCH calls.

warnings GenerationWarningType[]

Categories of warnings surfaced on this generated evaluator version. Present only on versions created via an EvaluatorGenerationJob when the paired job produced non-empty warnings. Absent (treat as no warnings) when the version is not from generation, when the paired job was clean, or when a subsequent PATCH to definition cleared the paired job's advisories. Follow generation_job_id to fetch the detailed warning payloads.

created_by string required

Creator of the evaluator

created_at string required

Creation date/time of the evaluator

modified_at string required

Last modified date/time of the evaluator

id string

Asset ID, a unique identifier for the asset

name string required

The name of the resource

version string required

The version of the resource

status string | enum required

Extensible status values shared by Foundry jobs.

Any of the following:

string

enum

queuedin_progresssucceededfailedcancelled

error OpenAI.Error

code string | null required

message string required

param string | null

type string

misalignment OpenAI.MisalignmentErrorDetailsResource

review_target string | null

error_type string | enum

Any of the following:

string

enum

potentially_unintended_data_transferpotentially_unintended_data_accesspotentially_unintended_destructive_activityother

detailed_explanation string

The public explanation for this block.

steer OpenAI._MisalignmentSteer

message string required

The public continuation instruction.

details OpenAI.Error[]

additionalInfo object

debugInfo object

sources EvaluatorGenerationJobSource[] required

Source materials for generation — agent descriptions, prompts, traces, or datasets. Each entry is an EvaluatorGenerationJobSource variant discriminated by type.

model string required

The LLM model to use for rubric generation (e.g., 'gpt-4o'). Required — users must provide their own model rather than relying on service-owned capacity.

evaluator_name string required

The evaluator name (immutable identifier). 1-256 characters; allowed characters are ASCII letters, digits, underscore (_), period (.), tilde (~), and hyphen (-). The prefix builtin. is reserved for system-managed evaluators and is rejected by the service. If an evaluator with this name already exists in the project (and is rubric-subtype), the service creates a new version under the same name and uses the prior version's dimensions as context for incremental improvement (foundation of the post-//build adaptive loop). Old versions remain queryable via get_version(name, version). If the existing evaluator is not a rubric-subtype evaluator (built-in, prompt-based, code-based), the request is rejected with 400 Bad Request.

evaluator_display_name string

Optional human-friendly display name for the resulting evaluator. Surfaced as EvaluatorVersion.display_name on the persisted evaluator. When omitted, the service uses evaluator_name as the display name. The evaluator_ prefix disambiguates this from the immutable evaluator_name identifier.

evaluator_description string

Optional human-friendly description for the resulting evaluator. Surfaced as EvaluatorVersion.description on the persisted evaluator. Typically collected from the UI alongside evaluator_display_name. The evaluator_ prefix disambiguates this from any other description fields on related models.

created_at integer required

finished_at integer

usage EvaluatorGenerationTokenUsage

Token consumption summary for an evaluator generation job. Populated when the job reaches a terminal state.

input_tokens integer required

Number of input (prompt) tokens consumed.

output_tokens integer required

Number of output (completion) tokens generated.

total_tokens integer required

Total tokens consumed (input + output).

input_quality_warnings RubricGenerationInputQualityWarning[]

Non-fatal input-quality advisories produced by the generation pipeline. Read-only; service-generated; populated only on terminal jobs when advisories fired. Omitted when generation was clean. Cleared when a subsequent PATCH to the paired EvaluatorVersion.definition invalidates the advisories.