Judge page

A judge scores runs of other agents. You’ll come to the Judge page to give a judge its first version, choose which agents it scores and when, and look back at what it has done. Select a judge from the Judges page to open it. Edit in the header changes its name, description, team, and owner.

The action menu holds Retire judge, Reinstate judge, and Delete judge. A retired judge takes no new runs; Prefactor only retires a judge whose runs have all finished.

Above the tabs, the deployment snapshot shows which version of the judge is deployed in each environment. A hosted judge’s version can move up to the next environment with Promote, after a confirmation, when that environment has an older version or none. A judge has no risk profile and no Activity schema tab. Only a remote judge has a Deployments tab, for the deployment tokens its own code uses, and a Setup judge item in the action menu that opens the getting started page.

What you can do here

Set up the judge with a criterion and a model, and update it later.
Choose which agents the judge scores, and whether it runs on demand or on completion.
Promote a version to the next environment.
Read the judge’s runs and see which run each one scored.
Check how the judge’s own runs perform.

Each of these has a CLI command and an API operation. Set up a judge from the CLI lists them next to the controls on this page.

Overview tab

The Overview tab holds the agents the judge scores and a summary of its recent work.

The Judging box lists the agents whose runs this judge scores, each with its trigger: On demand or On completion. Evals explains the difference. Add agent, in the box’s action menu, opens a dialog where you choose an agent and a trigger. It is unavailable when no agents are registered or every agent is already attached. A row’s action menu has Remove, which asks you to confirm before the judge stops scoring that agent.

Beside it, the Instances box counts the judge’s own runs by state, using the same states as the Agent page › Overview tab. Recent activity lists the five most recent judge runs, each naming the agent and run it scored, with View instance to open it.

Configuration tab

A hosted judge’s configuration is its list of versions. Pick a version from the Version dropdown to see the environments it is deployed in, its model, criterion, and system prompt template, and the recent runs on that version.

A new judge has no versions, so the tab shows Create judge. Once it has one, the button reads Update judge. Either opens a dialog with these fields:

A template picker: Write your own, or one of the built-in criteria — Completeness, Correctness, Hallucination, Refusal, Tool invocation, Tool response handling, PII detection, and Toxicity — each with a one-line description. Choosing a template fills in the criterion.
Criterion — what the judge looks for.
Model — one of the models from your LLM providers, grouped by provider. The dialog asks you to add a provider and a model first when you have none.
Version identifier — defaults to the current date and time.
Use a custom system prompt template — reveals System prompt template. The only variable is {{ criterion }}.

Update judge starts the dialog pre-filled from the selected version, and Start from a template brings the template picker back. Submitting saves a new version and deploys it to your lowest environment. If the judge already has a version with the same identifier, template, criterion, and model, Prefactor reuses it.

Existing versions never change. To change the criterion, template, or model, update the judge, which creates a new version.

History tab

Each row is one run of the judge, including runs still in progress. The columns are Time, Status, Instance ID, Environment, Version, and Alerts, with View to open the run. Under the instance ID, a “Judging” line names the agent and run that judge run scored and links to it.

Metrics tab

Run health for the judge over the time range you choose, with the same figures as the Agent page › Metrics tab.

Evals — the model behind judges, triggers, and judge runs.
LLM provider — the models a hosted judge runs on.
Judges page — the list of judges.
Set up a judge from the CLI — the same setup from the command line, including remote judges.
Agent page › Overview tab — where an agent’s own judges are added.
Agent › Instance › Quality tab — each judge’s verdict on a single run.