Set up a judge from the CLI

Each judge control in the web app has a matching CLI command, which suits scripted setups and CI. By the end of this page you have a hosted judge that scores an agent’s runs, a verdict on one run, and, if you need one, a remote judge whose verdicts come from your own code. For how judges work, see Evals.

Before you start

• The CLI installed and signed in with an account-scoped token. See Getting started with the CLI.
• An API key for Anthropic, OpenAI, Ollama Cloud, Together AI, or an OpenAI-compatible endpoint.
• An agent to score, with at least one finished run. Find the agent ID with prefactor agents list and a run ID with prefactor agent_instances list --agent_id <agent_id>.

Every command prints JSON. The IDs you need for the next step are in details.id.

Set up a hosted judge

1. Connect an LLM provider. Pass the key as @path so it stays out of your shell history:

   prefactor llm_providers create --name "Anthropic production" --provider anthropic --api_key @./anthropic-key.txt

   For a custom OpenAI-compatible endpoint, use --provider custom_openai and add --api_base_url https://llm.example.com/v1.

2. Add a model to the provider:

   prefactor llm_provider_models create --llm_provider_id <provider_id> --api_name claude-haiku-4-5

   For a named provider, the API name must be one models.dev lists for it, and Prefactor fills in the display name and pricing. For a custom provider, add --name, and --models_dev_provider with --models_dev_model_name if you want pricing.

3. Register the judge:

   prefactor judges create --name "Completeness judge"

4. Give the judge a version. To start from a built-in criterion, list the templates and save one’s criterion to a file:

   prefactor judges templates
   prefactor judges create_version --agent_id <judge_id> --hosted_llm_provider_model_id <model_id> --criterion @./criterion.md

   The version is deployed to your lowest environment. Add --system_prompt_template @./template.liquid to replace the instructions around the criterion; the only variable is {{ criterion }}.

5. Attach the judge to the agent it scores:

   prefactor agent_associations create --from_agent_id <judge_id> --to_agent_id <agent_id> --association_type judge --trigger on_demand

   Use --trigger on_completion to have the judge score every run that finishes as complete. Change it later with prefactor agent_associations update <association_id> --trigger on_completion.

6. Ask the judge to score a run:

   prefactor judges score <agent_instance_id> --judge_agent_id <judge_id>

   The response is the new judge run, in the Preparing state. Prefactor writes its input and starts it in the background.

Verify

Check the run you scored:

prefactor agent_instances retrieve <agent_instance_id>

Once the judge run finishes, details.quality_payloads has an entry named $judge:<judge_id> with a verdict and an explanation. The same verdict is on the run’s Agent › Instance › Quality tab. To see the judge run itself, list the judge’s runs with prefactor agent_instances list --agent_id <judge_id>.

Promote a judge version

A hosted judge runs in your highest environment with the version deployed furthest up. To move a version up one environment:

prefactor agent_deployments list_promotions --agent_id <judge_id>
prefactor agent_deployments promote <source_agent_deployment_id>

Run a remote judge

A remote judge’s logic runs in your own code. Prefactor creates a pending run for each request, and your code finishes it with the verdict.

1. Register the judge as remote:

   prefactor judges create --name "In-house policy judge" --execution_mode remote

2. Create a schema version whose input takes the run record and whose result is the verdict. Save the two schemas from Judge runs as input-schema.json and result-schema.json, then run:

   prefactor agent_schema_versions create --agent_id <judge_id> --external_identifier policy-judge-1 \
     --instance_params_payload_schema @./input-schema.json \
     --instance_result_payload_schema @./result-schema.json

3. Create a version on that schema version and deploy it to each environment whose runs the judge scores:

   prefactor agent_versions create --agent_id <judge_id> --external_identifier policy-judge-1 \
     --agent_schema_version_id <schema_version_id> --execution_mode remote
   prefactor agent_deployments create --agent_id <judge_id> --environment_id <environment_id> --current_version_id <version_id>

4. Attach it to an agent, as in step 5 above.

5. In your judge’s code, pick up pending runs, read the record from details.params_payload.message, and finish each run:

   prefactor agent_instances list --agent_id <judge_id> --statuses pending
   prefactor agent_instances retrieve <judge_run_id>
   prefactor agent_instances start <judge_run_id>
   prefactor agent_instances finish <judge_run_id> --status complete \
     --result_payload '{"verdict":"pass","explanation":"The reply answers both questions."}'

   Finish with --status failed when your code cannot score the run. A failed run copies nothing to the run it was asked to score.

Web app, CLI, and API

Each judge action has the same three ways in.

In the web app	CLI	API
Account › LLM providers tab: Create LLM provider, Edit, Delete	llm_providers create, update, delete	POST /api/v1/llm_provider, PUT, DELETE
LLM providers tab: Add model, model Edit, Delete	llm_provider_models create, update, delete	POST /api/v1/llm_provider_model, PUT, DELETE
Judges page: Register judge	judges create	POST /api/v1/agent with role: judge
Judge page: Edit, Retire judge, Reinstate judge, Delete judge	judges update, retire, reinstate, delete	PUT /api/v1/agent/{agent_id} and the agent retire, reinstate, and delete operations
Judge page › Configuration tab: Create judge, Update judge, template picker	judges create_version, judges templates	POST /api/v1/agent_version/hosted_judge, GET /api/v1/agent_version/judge_templates
Judge page: Promote	agent_deployments list_promotions, promote	GET /api/v1/agent_deployment/promotions, POST /api/v1/agent_deployment/{agent_deployment_id}/promote
Judge page › Overview tab Add agent, or Agent page › Overview tab Add judge; row Remove	agent_associations create, update, delete	POST /api/v1/agent_association, PUT, DELETE
Judging and Judges boxes	agent_associations list_from --agent_ids <judge_id>, list_to --agent_id <agent_id>	GET /api/v1/agent_association/from, GET /api/v1/agent_association/to
Agent › Instance › Quality tab: Run judge	judges score	POST /api/v1/agent_instance/{agent_instance_id}/judge
Quality tab View run, Judge page › History tab “Judging” line	agent_instance_associations list_from, list_to	GET /api/v1/agent_instance_association/from and /to
No web app control: a remote judge’s own code	agent_instances list --statuses pending, start, finish --result_payload	GET /api/v1/agent_instance, start, finish

If it didn’t work

• api_name “is not a known model for this provider”. For a named provider, the model must be one models.dev lists. Check the spelling against the provider’s model names, or use a custom provider.
• Register judge is unavailable in the web app. The account has no LLM provider with a model. Run steps 1 and 2.
• “This agent does not judge that agent instance”. The judge is not attached to the agent that owns that run. Run step 5.
• “The judge has no versions”. Run step 4.
• “The judge has no deployment in the instance’s environment” or “The judge deployment has no current version”. A remote judge needs a deployment with a current version in the environment of the run it scores. Run step 3 of the remote judge setup for that environment.
• “The judge’s current version needs an instance params schema and an instance result schema”. The remote judge’s version has no input or result schema. Create a schema version with both, then a new version on it.
• The judge run failed. For a hosted judge, check the provider key with prefactor llm_providers update <provider_id> --api_key @./key.txt; Prefactor does not test the key when you save it. For a remote judge, check that the result has a verdict of pass or fail and an explanation.

Related

• Evals — judges, triggers, judge runs, and where verdicts land.
• LLM provider — providers, models, and keys.
• Judge page — the same setup in the web app.
• Record a run’s input, result, and metadata — the payloads a remote judge reads and writes.