Evals

A trace tells you what an agent did, but not whether its answer was any good, and an agent soon handles more runs than anyone can read. You can produce quality evaluations yourself and attach them to each run. Evals produce them inside Prefactor instead: a judge reads another agent’s run against a criterion and returns a verdict, which Prefactor records on that run.

A judge is an agent

A judge is an agent with the role judge. Every agent has a role, either standard (the default) or judge. You choose it when you register the agent, and it cannot change afterwards. Apart from the role, a judge is an ordinary agent: it has versions, deployments, and instances, and it belongs to your account like any other. The web app lists judges in their own Judges section, separate from Agents but grouped by team in the same way.

Because a judge is an agent, its work is recorded the way any agent’s work is. Each time it scores a run, that is an instance of the judge, with a start, a finish, and a result.

Who scores whom

A judge and the agents it scores are linked many-to-many. One judge can score any number of agents, and one agent can have any number of judges — a correctness judge and a tone judge on the same support agent, for example. The scored side can be any agent, including another judge, which is how you check a judge’s own verdicts. An agent cannot be linked to itself, and both agents belong to the same account. Scoring a single run of the scored agent is the only thing a link does today.

The link also carries the trigger, which decides when the judge runs.

Hosted and remote judges

Where a judge runs is the other choice you make when you register it. A hosted judge runs on Prefactor: Prefactor sends the run to a language model from one of your LLM providers, and the model returns the verdict. A remote judge runs in your own infrastructure: Prefactor prepares the work and waits for your code to pick it up and report back. Judges you register in the web app are hosted. To register a remote judge, set the agent’s execution mode to remote from the CLI or the API; Set up a judge from the CLI covers both kinds.

A hosted judge’s configuration lives on its version. It has three parts: a criterion, which says what the judge is looking for; a system prompt template that wraps the criterion; and the model that runs it. Prefactor ships built-in criteria for completeness, correctness, hallucination, refusal, tool invocation, tool response handling, PII detection, and toxicity. You can start from one of them or write your own, and you can replace the surrounding template. The only variable a template can use is criterion.

A hosted judge answers in a single turn, and Prefactor leaves the response length at the model’s maximum. Changing the criterion, the template, or the model creates a new version rather than editing the old one, so every verdict traces back to the configuration that produced it.

Environments behave differently for hosted judges than for other agents. A new hosted judge version starts on your lowest environment. When the judge is asked to score a run, it runs in your highest environment, whatever environment the scored run was in, using the version deployed furthest up. If no environment has a deployment, it uses the judge’s newest version. Asking a judge pins the version it used on the highest environment, so a version you never promoted is deployed there once it has run. You move a version up one environment at a time by promoting it. A remote judge’s run stays in the environment of the run it scores, so a remote judge needs a deployment with a current version in that environment.

Triggers

By default a link is on demand: the judge scores a run only when someone asks, from the instance’s Quality tab, with prefactor judges score, or through the API. A link set to on completion asks the judge every time a run of the scored agent finishes as complete. Runs that fail, are cancelled, or are terminated do not trigger it. Both kinds of judge can use on completion: a hosted judge starts on its own, and a remote judge gets a pending run to pick up.

If a judge cannot run when a run completes — your account has no environments, the judge has no version deployed where it needs one, its version does not declare an input and a result schema, or its hosted version is not single-turn — Prefactor skips it and the run still finishes. Asking the same judge on demand returns an error instead.

Judge runs

Each time a judge scores a run, Prefactor creates an instance of the judge, called a judge run. It appears in the judge’s history and links back to the run it scored. A judge run starts as Preparing while Prefactor writes its input, then moves to Pending. A hosted judge run starts straight away; a remote one waits in Pending for your code.

The judge’s input is a record that Prefactor writes of the scored run: the agent and version, the run’s status and times, its input, result, and metadata, and then what happened during it — the conversation when the run can be shown as one, otherwise the spans. The record is stored as the judge run’s input. Prefactor does not score a run whose record would exceed 5,000 spans or 250,000 characters.

A hosted judge sends the record to the model as the user message, with the criterion in the system prompt, and asks for structured output: a verdict of pass or fail, and an explanation. The model’s answer becomes the judge run’s result. If the answer does not fit that shape, the judge run fails.

A remote judge does the same job through the API or the CLI; Run a remote judge has the commands. Your code lists the judge’s pending instances, reads the record from the run’s input (a message field), starts the run, and finishes it with the verdict as the result. To have the API accept those payloads, the judge’s schema version declares an input schema that takes message and a result schema for the verdict:

{
  "instance_params_payload_schema": {
    "title": "Run to score",
    "schema": {
      "type": "object",
      "properties": { "message": { "type": "string" } },
      "required": ["message"]
    }
  },
  "instance_result_payload_schema": {
    "title": "Judgement",
    "schema": {
      "type": "object",
      "properties": {
        "verdict": { "type": "string", "enum": ["pass", "fail"] },
        "explanation": { "type": "string" }
      },
      "required": ["verdict", "explanation"]
    }
  }
}

A remote judge that finishes as complete sends a result such as {"verdict": "pass", "explanation": "The reply answers both questions."}. One that cannot score the run finishes as failed, and a failed finish does not need a result. See the instance schemas and payloads behind this on the instance page.

Where the verdict lands

When a judge run finishes as complete with a result, Prefactor copies the result onto the run it scored, as a quality payload named $judge: followed by the judge’s agent ID. Each judge gets its own payload, so two judges on one agent never overwrite each other. Prefactor also records the change as a quality span, as it does for any quality payload. Nothing needs declaring on the agent being scored: Prefactor displays judge results with a built-in schema that shows the verdict and explanation under the judge’s name.

The payload belongs to the scored run, so it stays if the judge is deleted. A judge run that fails copies nothing. Names that start with $ are reserved for Prefactor, so your own quality schemas cannot use one. A judge’s result cannot carry sensitive markers; Prefactor rejects a finish that does.