EvaluatorGenerationInputs Class

Caller-supplied inputs for an evaluator generation job.

Constructor

EvaluatorGenerationInputs(*args: Any, **kwargs: Any)

Variables

Name	Description
sources	list[EvaluatorGenerationJobSource] Source materials for generation — agent descriptions, prompts, traces, or datasets. Each entry is an `EvaluatorGenerationJobSource` variant discriminated by `type`. Required.
model	str The LLM model to use for rubric generation (e.g., 'gpt-4o'). Required — users must provide their own model rather than relying on service-owned capacity. Required.
evaluator_name	str The evaluator name (immutable identifier). 1-256 characters; allowed characters are ASCII letters, digits, underscore (`_`), period (`.`), tilde (`~`), and hyphen (`-`). The prefix `builtin.` is reserved for system-managed evaluators and is rejected by the service. If an evaluator with this name already exists in the project (and is rubric-subtype), the service creates a new version under the same name and uses the prior version's `dimensions` as context for incremental improvement (foundation of the post-//build adaptive loop). Old versions remain queryable via `get_version(name, version)`. If the existing evaluator is not a rubric-subtype evaluator (built-in, prompt-based, code-based), the request is rejected with `400 Bad Request`. Required.
evaluator_display_name	str Optional human-friendly display name for the resulting evaluator. Surfaced as `EvaluatorVersion.display_name` on the persisted evaluator. When omitted, the service uses `evaluator_name` as the display name. The `evaluator_` prefix disambiguates this from the immutable `evaluator_name` identifier.
evaluator_description	str Optional human-friendly description for the resulting evaluator. Surfaced as `EvaluatorVersion.description` on the persisted evaluator. Typically collected from the UI alongside `evaluator_display_name`. The `evaluator_` prefix disambiguates this from any other description fields on related models.

Methods

as_dict	Return a dict that can be turned into json using json.dump.
clear	Remove all items from D.
copy
get	Get the value for key if key is in the dictionary, else default. :param str key: The key to look up. :param any default: The value to return if key is not in the dictionary. Defaults to None :returns: D[k] if k in D, else d. :rtype: any
items
keys
pop	Removes specified key and return the corresponding value. :param str key: The key to pop. :param any default: The value to return if key is not in the dictionary :returns: The value corresponding to the key. :rtype: any :raises KeyError: If key is not found and default is not given.
popitem	Removes and returns some (key, value) pair :returns: The (key, value) pair. :rtype: tuple :raises KeyError: if D is empty.
setdefault	Same as calling D.get(k, d), and setting D[k]=d if k not found :param str key: The key to look up. :param any default: The value to set if key is not in the dictionary :returns: D[k] if k in D, else d. :rtype: any
update	Updates D from mapping/iterable E and F. :param any args: Either a mapping object or an iterable of key-value pairs.
values

as_dict

Return a dict that can be turned into json using json.dump.

as_dict(*, exclude_readonly: bool = False) -> dict[str, Any]

Keyword-Only Parameters

Name	Description
exclude_readonly	bool Whether to remove the readonly properties. Default value: False

Returns

Type	Description
dict	A dict JSON compatible object

clear

Remove all items from D.

clear() -> None

copy

copy() -> Model

get

Get the value for key if key is in the dictionary, else default. :param str key: The key to look up. :param any default: The value to return if key is not in the dictionary. Defaults to None :returns: D[k] if k in D, else d. :rtype: any

get(key: str, default: Any = None) -> Any

Parameters

Name	Description
key Required
default	Default value: None

items

items() -> ItemsView[str, Any]

Returns

Type	Description
ItemsView	set-like object providing a view on D's items

keys

keys() -> KeysView[str]

Returns

Type	Description
KeysView	a set-like object providing a view on D's keys

pop

Removes specified key and return the corresponding value. :param str key: The key to pop. :param any default: The value to return if key is not in the dictionary :returns: The value corresponding to the key. :rtype: any :raises KeyError: If key is not found and default is not given.

pop(key: str, default: ~typing.Any = <object object>) -> Any

Parameters

Name	Description
key Required
default

popitem

Removes and returns some (key, value) pair :returns: The (key, value) pair. :rtype: tuple :raises KeyError: if D is empty.

popitem() -> tuple[str, Any]

setdefault

Same as calling D.get(k, d), and setting D[k]=d if k not found :param str key: The key to look up. :param any default: The value to set if key is not in the dictionary :returns: D[k] if k in D, else d. :rtype: any

setdefault(key: str, default: ~typing.Any = <object object>) -> Any

Parameters

Name	Description
key Required
default

update

Updates D from mapping/iterable E and F. :param any args: Either a mapping object or an iterable of key-value pairs.

update(*args: Any, **kwargs: Any) -> None

values

values() -> ValuesView[Any]

Returns

Type	Description
ValuesView	an object providing a view on D's values

Attributes

evaluator_description

Optional human-friendly description for the resulting evaluator. Surfaced as EvaluatorVersion.description on the persisted evaluator. Typically collected from the UI alongside evaluator_display_name. The evaluator_ prefix disambiguates this from any other description fields on related models.

evaluator_description: str | None

evaluator_display_name

Optional human-friendly display name for the resulting evaluator. Surfaced as EvaluatorVersion.display_name on the persisted evaluator. When omitted, the service uses evaluator_name as the display name. The evaluator_ prefix disambiguates this from the immutable evaluator_name identifier.

evaluator_display_name: str | None

evaluator_name

The evaluator name (immutable identifier). 1-256 characters; allowed characters are ASCII letters, digits, underscore (_), period (.), tilde (~), and hyphen (-). The prefix builtin. is reserved for system-managed evaluators and is rejected by the service. If an evaluator with this name already exists in the project (and is rubric-subtype), the service creates a new version under the same name and uses the prior version's dimensions as context for incremental improvement (foundation of the post-//build adaptive loop). Old versions remain queryable via get_version(name, version). If the existing evaluator is not a rubric-subtype evaluator (built-in, prompt-based, code-based), the request is rejected with 400 Bad Request. Required.

evaluator_name: str

model

The LLM model to use for rubric generation (e.g., 'gpt-4o'). Required — users must provide their own model rather than relying on service-owned capacity. Required.

model: str

sources

Source materials for generation — agent descriptions, prompts, traces, or datasets. Each entry is an EvaluatorGenerationJobSource variant discriminated by type. Required.

sources: list['_models.EvaluatorGenerationJobSource']

Feedback

Was this page helpful?