---
title: Set up a judge from the CLI
description: Connect a model, register a hosted or remote judge, attach it to an
  agent, and score a run with the Prefactor CLI, with the matching web app
  controls and API operations.
editUrl: true
head: []
template: doc
sidebar:
  hidden: false
  attrs: {}
pagefind: true
draft: false
---

Each judge control in the web app has a matching CLI command, which suits scripted setups and CI. By the end of this page you have a hosted judge that scores an agent's runs, a verdict on one run, and, if you need one, a remote judge whose verdicts come from your own code. For how judges work, see [Evals](/platform/evals).

## Before you start

- The CLI installed and signed in with an account-scoped token. See [Getting started with the CLI](/cli/getting-started).
- An API key for Anthropic, OpenAI, Ollama Cloud, Together AI, or an OpenAI-compatible endpoint.
- An agent to score, with at least one finished run. Find the agent ID with [`prefactor agents list`](/cli) and a run ID with [`prefactor agent_instances list --agent_id <agent_id>`](/cli).

Every command prints JSON. The IDs you need for the next step are in `details.id`.

## Set up a hosted judge

1. Connect an LLM provider. Pass the key as `@path` so it stays out of your shell history:

   ```bash
   prefactor llm_providers create --name "Anthropic production" --provider anthropic --api_key @./anthropic-key.txt
   ```

   For a custom OpenAI-compatible endpoint, use `--provider custom_openai` and add `--api_base_url https://llm.example.com/v1`.

2. Add a model to the provider:

   ```bash
   prefactor llm_provider_models create --llm_provider_id <provider_id> --api_name claude-haiku-4-5
   ```

   For a named provider, the API name must be one models.dev lists for it, and Prefactor fills in the display name and pricing. For a custom provider, add `--name`, and `--models_dev_provider` with `--models_dev_model_name` if you want pricing.

3. Register the judge:

   ```bash
   prefactor judges create --name "Completeness judge"
   ```

4. Give the judge a version. To start from a built-in criterion, list the templates and save one's `criterion` to a file:

   ```bash
   prefactor judges templates
   prefactor judges create_version --agent_id <judge_id> --hosted_llm_provider_model_id <model_id> --criterion @./criterion.md
   ```

   The version is deployed to your lowest environment. Add `--system_prompt_template @./template.liquid` to replace the instructions around the criterion; the only variable is `{{ criterion }}`.

5. Attach the judge to the agent it scores:

   ```bash
   prefactor agent_associations create --from_agent_id <judge_id> --to_agent_id <agent_id> --association_type judge --trigger on_demand
   ```

   Use `--trigger on_completion` to have the judge score every run that finishes as complete. Change it later with [`prefactor agent_associations update <association_id> --trigger on_completion`](/cli).

6. Ask the judge to score a run:

   ```bash
   prefactor judges score <agent_instance_id> --judge_agent_id <judge_id>
   ```

   The response is the new judge run, in the Preparing state. Prefactor writes its input and starts it in the background.

## Verify

Check the run you scored:

```bash
prefactor agent_instances retrieve <agent_instance_id>
```

Once the judge run finishes, `details.quality_payloads` has an entry named `$judge:<judge_id>` with a `verdict` and an `explanation`. The same verdict is on the run's [Agent › Instance › Quality tab](/admin-ui/agent-instance/quality). To see the judge run itself, list the judge's runs with [`prefactor agent_instances list --agent_id <judge_id>`](/cli).

## Promote a judge version

A hosted judge runs in your highest environment with the version deployed furthest up. To move a version up one environment:

```bash
prefactor agent_deployments list_promotions --agent_id <judge_id>
prefactor agent_deployments promote <source_agent_deployment_id>
```

## Run a remote judge

A remote judge's logic runs in your own code. Prefactor creates a pending run for each request, and your code finishes it with the verdict.

1. Register the judge as remote:

   ```bash
   prefactor judges create --name "In-house policy judge" --execution_mode remote
   ```

2. Create a schema version whose input takes the run record and whose result is the verdict. Save the two schemas from [Judge runs](/platform/evals#judge-runs) as `input-schema.json` and `result-schema.json`, then run:

   ```bash
   prefactor agent_schema_versions create --agent_id <judge_id> --external_identifier policy-judge-1 \
     --instance_params_payload_schema @./input-schema.json \
     --instance_result_payload_schema @./result-schema.json
   ```

3. Create a version on that schema version and deploy it to each environment whose runs the judge scores:

   ```bash
   prefactor agent_versions create --agent_id <judge_id> --external_identifier policy-judge-1 \
     --agent_schema_version_id <schema_version_id> --execution_mode remote
   prefactor agent_deployments create --agent_id <judge_id> --environment_id <environment_id> --current_version_id <version_id>
   ```

4. Attach it to an agent, as in step 5 above.

5. In your judge's code, pick up pending runs, read the record from `details.params_payload.message`, and finish each run:

   ```bash
   prefactor agent_instances list --agent_id <judge_id> --statuses pending
   prefactor agent_instances retrieve <judge_run_id>
   prefactor agent_instances start <judge_run_id>
   prefactor agent_instances finish <judge_run_id> --status complete \
     --result_payload '{"verdict":"pass","explanation":"The reply answers both questions."}'
   ```

   Finish with `--status failed` when your code cannot score the run. A failed run copies nothing to the run it was asked to score.

## Web app, CLI, and API

Each judge action has the same three ways in.

| In the web app | CLI | API |
| --- | --- | --- |
| [Account › LLM providers tab](/admin-ui/account/llm-providers): **Create LLM provider**, **Edit**, **Delete** | `llm_providers create`, `update`, `delete` | [POST /api/v1/llm_provider](/api/platform/operations/actionllmprovidercreate), [PUT](/api/platform/operations/actionllmproviderupdate), [DELETE](/api/platform/operations/actionllmproviderdelete) |
| LLM providers tab: **Add model**, model **Edit**, **Delete** | `llm_provider_models create`, `update`, `delete` | [POST /api/v1/llm_provider_model](/api/platform/operations/actionllmprovidermodelcreate), [PUT](/api/platform/operations/actionllmprovidermodelupdate), [DELETE](/api/platform/operations/actionllmprovidermodeldelete) |
| [Judges page](/admin-ui/judges): **Register judge** | `judges create` | [POST /api/v1/agent](/api/platform/operations/actionagentcreate) with `role: judge` |
| [Judge page](/admin-ui/judge): **Edit**, **Retire judge**, **Reinstate judge**, **Delete judge** | `judges update`, `retire`, `reinstate`, `delete` | [PUT /api/v1/agent/{agent_id}](/api/platform/operations/actionagentupdate) and the agent retire, reinstate, and delete operations |
| [Judge page › Configuration tab](/admin-ui/judge#configuration-tab): **Create judge**, **Update judge**, template picker | `judges create_version`, `judges templates` | [POST /api/v1/agent_version/hosted_judge](/api/platform/operations/actionagentversioncreatehostedjudge), [GET /api/v1/agent_version/judge_templates](/api/platform/operations/queryagentversionlistjudgetemplates) |
| Judge page: **Promote** | `agent_deployments list_promotions`, `promote` | [GET /api/v1/agent_deployment/promotions](/api/platform/operations/queryagentdeploymentlistpromotions), [POST /api/v1/agent_deployment/{agent_deployment_id}/promote](/api/platform/operations/actionagentdeploymentpromote) |
| [Judge page › Overview tab](/admin-ui/judge#overview-tab) **Add agent**, or [Agent page › Overview tab](/admin-ui/agent/overview#judges) **Add judge**; row **Remove** | `agent_associations create`, `update`, `delete` | [POST /api/v1/agent_association](/api/platform/operations/actionagentassociationcreate), [PUT](/api/platform/operations/actionagentassociationupdate), [DELETE](/api/platform/operations/actionagentassociationdelete) |
| Judging and Judges boxes | `agent_associations list_from --agent_ids <judge_id>`, `list_to --agent_id <agent_id>` | [GET /api/v1/agent_association/from](/api/platform/operations/queryagentassociationlistfrom), [GET /api/v1/agent_association/to](/api/platform/operations/queryagentassociationlistto) |
| [Agent › Instance › Quality tab](/admin-ui/agent-instance/quality): **Run judge** | `judges score` | [POST /api/v1/agent_instance/{agent_instance_id}/judge](/api/platform/operations/actionagentinstancejudge) |
| Quality tab **View run**, Judge page › History tab "Judging" line | `agent_instance_associations list_from`, `list_to` | GET /api/v1/agent_instance_association/from and /to |
| No web app control: a remote judge's own code | `agent_instances list --statuses pending`, `start`, `finish --result_payload` | [GET /api/v1/agent_instance](/api/platform/operations/queryagentinstancelist), [start](/api/platform/operations/actionagentinstancestart), [finish](/api/platform/operations/actionagentinstancefinish) |

## If it didn't work

- **`api_name` "is not a known model for this provider".** For a named provider, the model must be one models.dev lists. Check the spelling against the provider's model names, or use a custom provider.
- **Register judge is unavailable in the web app.** The account has no LLM provider with a model. Run steps 1 and 2.
- **"This agent does not judge that agent instance".** The judge is not attached to the agent that owns that run. Run step 5.
- **"The judge has no versions".** Run step 4.
- **"The judge has no deployment in the instance's environment" or "The judge deployment has no current version".** A remote judge needs a deployment with a current version in the environment of the run it scores. Run step 3 of the remote judge setup for that environment.
- **"The judge's current version needs an instance params schema and an instance result schema".** The remote judge's version has no input or result schema. Create a schema version with both, then a new version on it.
- **The judge run failed.** For a hosted judge, check the provider key with `prefactor llm_providers update <provider_id> --api_key @./key.txt`; Prefactor does not test the key when you save it. For a remote judge, check that the result has a `verdict` of `pass` or `fail` and an `explanation`.

## Related

- [Evals](/platform/evals) — judges, triggers, judge runs, and where verdicts land.
- [LLM provider](/platform/concepts/llm-provider) — providers, models, and keys.
- [Judge page](/admin-ui/judge) — the same setup in the web app.
- [Record a run's input, result, and metadata](/api/instance-payloads) — the payloads a remote judge reads and writes.