Skip to content
PrefactorPrefactorPrefactor Docs

Quality tab

The quality evaluations for the run, one section per judge or quality schema, so you can see how the run was scored and why.

Evaluations reach a run in two ways. Your own evaluation process submits them after the run finished, through the SDK or API, shaped by the quality schemas declared in the agent’s activity schema. And the judges attached to the agent score the run themselves. Each change to a named payload is recorded as a quality span, visible on the Activity tab.

For what sits above the tab row, see the Agent instance page.

Judges

Section titled “Judges”

Judge sections come first, one for each judge attached to the run’s agent, in order of the judge’s name. Each section is headed by the judge’s name. Once the judge has scored the run, the name of its verdict payload sits underneath: $judge: followed by the judge’s agent ID. Next come any of the judge’s runs on this instance that are still Preparing, Pending, or Active, each with a View run link and a status badge, so you can follow a verdict that is still being produced. The verdict follows as a one-line summary, such as “Pass: The reply answers both questions.”, then a Measure and Result table with a pass or fail badge and the explanation.

A judge that hasn’t scored the run yet shows a note instead of a verdict:

  • A judge set to run on demand says it hasn’t scored this instance and offers Run judge.
  • A judge set to run on completion says “This judge runs when this instance finishes.” while the run is still going. Once the run has ended without a verdict, it says the judge runs when an instance finishes and has not scored this one.

Every judge section also has Run judge in its action menu, whatever the trigger. Use it to ask a judge again after it has scored the run, or to have an on-completion judge score a run it did not pick up, such as one that failed. From the command line, the same request is prefactor judges score <agent_instance_id> --judge_agent_id <judge_id>.

Which judges appear, and their triggers, are set on the Agent page › Overview tab and the Judge page.

Other quality payloads

Section titled “Other quality payloads”

After the judges, each other named quality payload the run recorded gets its own section, in order of payload name. The heading is the schema’s title, or the payload name when no title is set, with the payload name shown underneath. If the schema has a summary template, the rendered one-line summary comes next, then the payload as a Measure and Result table labelled from the schema.

A verdict from a judge that is no longer attached to the agent also lands here, in a section of its own. It is headed “Judgement” if the judge has been deleted.

Section titled “Related”