Find or create a hosted judge version

POST

/api/v1/agent_version/hosted_judge

const url = 'https://app.prefactorai.com/api/v1/agent_version/hosted_judge';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"agent_id":"013xrzp12g3nqk8ns6jadrqmpgrjkeny","criterion":"Decide whether the agent completed every request.","external_identifier":"2026-10-01 15:03","hosted_llm_provider_model_id":"013xrzp12g3nqk8ns6jadrqmpgrjkeny","id":"013xrzp12g3nqk8ns6jadrqmpgrjkeny","system_prompt_template":"You are a judge.\n\n{{ criterion }}\n\nAnswer with a verdict."}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request POST \
  --url https://app.prefactorai.com/api/v1/agent_version/hosted_judge \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "agent_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny", "criterion": "Decide whether the agent completed every request.", "external_identifier": "2026-10-01 15:03", "hosted_llm_provider_model_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny", "id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny", "system_prompt_template": "You are a judge.\n\n{{ criterion }}\n\nAnswer with a verdict." }'

Returns the matching hosted judge version if one already exists for this judge with the same external identifier, system prompt template, criterion, and model. Otherwise creates a new version. The version is hosted and single-turn, uses the canonical judge schema, and records Prefactor’s runtime environment. hosted_configuration.system_prompt stores the template and hosted_configuration.judge_criterion stores the rubric. The runtime renders them when a run starts. The returned version is deployed to the lowest environment.

Authorizations

authorization

Request Body

AgentVersion_DetailsForCreateHostedJudge

object

agent_id

required

Agent_Id

A valid agent ID

string

Example

013xrzp12g3nqk8ns6jadrqmpgrjkeny

criterion

required

The rubric this judge applies. Stored as hosted_configuration.judge_criterion and rendered into the system prompt when a run starts

string

Example

Decide whether the agent completed every request.

external_identifier

One of:

string
Null

string

Example

2026-10-01 15:03

hosted_llm_provider_model_id

required

LlmProviderModel_Id

A valid LLM provider model ID

string

Example

013xrzp12g3nqk8ns6jadrqmpgrjkeny

One of:

AgentVersion_Id
Null

AgentVersion_Id

A valid agent version ID

string

Example

013xrzp12g3nqk8ns6jadrqmpgrjkeny

system_prompt_template

One of:

string
Null

string

Example

You are a judge.

{{ criterion }}

Answer with a verdict.

Responses

200

Success

application/json

object

details

required

AgentVersion_Details

object

account_id

required

Account_Id

A valid account ID

string

agent_id

required

Agent_Id

A valid agent ID

string

agent_schema_version_id

required

AgentSchemaVersion_Id

A valid agent schema version ID

string

current_agent_deployment_refs

required

Agent deployments that pin this version as their current version (empty if none).

Array<string>

execution_mode

required

Whether Prefactor executes this version (hosted) or an external agent reports telemetry (remote)

string

Allowed values: hosted remote

external_identifier

required

External identifier for the agent version

string

external_identifier_repeats

required

Count of rows sharing this external identifier (helps distinguish duplicates).

integer

hosted_configuration

required

One of:

AgentVersion_HostedConfiguration
Null

AgentVersion_HostedConfiguration

object

judge_criterion

Criterion rendered into the system prompt when a hosted run starts. Omitted when the system prompt is used as written

string

llm_provider_model_id

required

LlmProviderModel_Id

A valid LLM provider model ID

string

mode

required

Hosted run mode: one model turn, or a conversation

string

Allowed values: single_turn conversation

system_prompt

required

System prompt for each hosted run. When judge_criterion is set, this is a Liquid template and criterion is its only variable

string

required

AgentVersion_Id

A valid agent version ID

string

inserted_at

required

When the agent version was created

string format: date-time

observed_classification

One of:

string
Null

string

Allowed values: unknown public internal confidential restricted secret

runtime_environment

required

One of:

AgentVersion_RuntimeEnvironment
Null

AgentVersion_RuntimeEnvironment

object

agent_sdk

One of:

Array<string>
Null

List of package name and version strings (e.g. “package@1.0.0”)

Array<string>

One of:

string
Null

Operating system (e.g. “linux”)

string

prefactor_sdk

One of:

Array<string>
Null

List of package name and version strings (e.g. “package@1.0.0”)

Array<string>

runtime

One of:

string
Null

Language runtime version (e.g. “python@3.12”)

string

type

required

Type of resource (always ‘agent_version’)

string

Allowed values: agent_version

updated_at

required

When the agent version was last updated

string format: date-time

status

required

string

Allowed values: success

Example

{
  "details": {
    "account_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "agent_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "agent_schema_version_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "current_agent_deployment_refs": [
      "013xrzp12g3nqk8ns6jadrqmpgrjkeny"
    ],
    "execution_mode": "hosted",
    "external_identifier": "v1.0.0",
    "hosted_configuration": null,
    "id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "inserted_at": "2024-01-01T00:00:00Z",
    "observed_classification": "unknown",
    "runtime_environment": null,
    "type": "agent_version",
    "updated_at": "2024-01-01T00:00:00Z"
  },
  "status": "success"
}

default

Error

application/json

One of:

object

code

required

string

Allowed values: bad_request bad_authtoken not_authenticated not_permitted not_implemented unknown not_found unexpected_ref_type unexpected invalid_action invalid_state alert_unknown alert_already_cleared invalid_value required_value idempotency_key_already_used temporarily_unavailable conflict

message

required

string

status

required

string

Allowed values: error

Example

{
  "code": "bad_request",
  "status": "error"
}