Skip to content

Quality and performance

Risk tells you which declared data categories and actions a run exercised (from the activity schema and span-type counts). Quality and performance tell you whether the agent actually did its job well. These are different questions: a governance review asks the first, and a business review asks the second. An agent can score well on one while failing the other.

Quality for an AI agent is whether it produces outputs that are accurate, relevant, and useful given what was asked of it. This is harder to define than correctness in traditional software, where a function either returns the right result or it does not. An agent’s outputs are often natural language, judgement calls, or sequences of decisions that only make sense in context, and evaluating them requires knowing what “right” looks like for that task.

Reliability is part of quality: whether runs complete, whether tool calls succeed, whether the agent stays on task. These are the technical foundations on which usefulness depends. An agent that fails half its runs has a quality problem, not a performance problem. Prefactor summarises these signals per agent on the agent’s Metrics tab: success rates, duration distributions, and failure counts over a time range you choose.

The harder half of quality, whether an output was actually good, cannot be read off a trace, because it depends on knowing what the task was. Prefactor does not decide what good looks like for you. You define what a quality evaluation looks like for your agent and evaluate runs however suits you: with your own process, which attaches the result to each run, or with a judge, an agent you register that scores runs against criteria you write and records its verdict on the run.

Because “good” means something different for every agent, the shape of a quality evaluation is yours to define. An agent’s activity schema can include one or more named quality schemas. Each is a JSON Schema describing an evaluation payload for that agent’s runs (a score and a verdict, a set of rubric fields, whatever your evaluation produces), plus an optional template that renders the payload as a one-line summary. Naming lets one agent carry several kinds of evaluation at once (a summary-quality schema and a policy-compliance schema, for example), evaluated and recorded independently.

Unless a judge does the work, the evaluation itself happens outside Prefactor, and usually after the run: an automated eval suite, a model grading the output, or a person reviewing it. Whatever produces the judgement, it attaches the result to the instance as a quality payload for the matching schema name, through the SDK or the API. Prefactor renders the summary from that schema’s template and shows both on the instance’s Quality tab.

A judge’s verdict is a quality payload too, under a reserved name of the form $judge: followed by the judge’s agent ID. Evals explains how judges, the agents they score, and their triggers fit together.

Quality payloads are part of the audit trail, not just a mutable field. Every time a named quality payload is set, changed, or cleared, Prefactor records the change as a span within that instance. This quality span is written by the platform itself and kept apart from the agent’s own activity. The evaluation history of a run is reconstructable from its span record, the same way the run itself is.

Once you evaluate runs routinely, not every run is production traffic. Each instance records a purpose: Live for a real run, Eval for an evaluation run, or Smoke test for a pipeline check. Your integration sets the purpose when the run is registered; it defaults to Live. The purpose is shown wherever the instance appears, so evaluation traffic is distinguishable from the traffic it is meant to protect.

Performance is whether the agent, as a whole, is delivering the business value it was put in place to deliver. The question is not whether any individual output was correct, but whether the initiative is working.

Prefactor does not compute a business performance score. What it provides is a continuous run history (and the quality evaluations attached to it) against which you assess change outside the product: across many runs, over time, against the outcomes the agent was intended to drive.

Agents change for reasons that are not always obvious or deliberate. The underlying model may be updated by the provider. Code or prompts may be revised. The way people interact with the agent shifts as it becomes familiar. Any of these can affect quality and performance, and without a continuous record, it is hard to know whether something changed, when it changed, or what caused it.

Prefactor’s run history (and the quality evaluations attached to it) gives you a continuous record against which change becomes visible. When something shifts, for better or worse, you have the before and after to compare.

  • Evals — judges that score runs for you, and where their verdicts end up.
  • Record quality evaluations — declare quality schemas and submit payloads from the SDKs.
  • Agent page › Metrics tab — success, duration, and failure signals for an agent over a time range you choose.
  • Agent › Instance › Quality tab — the rendered evaluation for a single run.
  • Risk — the separate question of declared capabilities exercised in a run.
  • Instance — the run-level record, including purpose, lifecycle state, and version.