Quality and performance

Risk tells you which declared data categories and actions a run exercised (from the activity schema and span-type counts). Quality and performance tell you whether the agent actually did its job well. These are different questions — a governance review asks the first; a business review asks the second — and an agent can score well on one while failing the other.

Quality

Quality for an AI agent is whether it produces outputs that are accurate, relevant, and useful given what was asked of it. This is harder to define than correctness in traditional software, where a function either returns the right result or it does not. An agent’s outputs are often natural language, judgement calls, or sequences of decisions that only make sense in context — and evaluating them requires knowing what “right” looks like for that task.

Reliability is part of quality: whether runs complete, whether tool calls succeed, whether the agent stays on task. These are the technical foundations on which usefulness depends. An agent that fails half its runs is not a performance problem — it is a quality problem. Prefactor summarises these signals per agent — success rates, duration distributions, and failure counts over the last day — on the agent’s Quality tab.

The harder half of quality — whether an output was actually good — cannot be read off a trace, because it depends on knowing what the task was. Prefactor does not judge your agent’s outputs; its role is to record the judgement you make. You define what a quality evaluation looks like for your agent, evaluate runs however suits you, and attach the result to each run.

Recording quality evaluations

Because “good” means something different for every agent, the shape of a quality evaluation is yours to define. An agent’s activity schema can include one or more named quality schemas — each a JSON Schema describing an evaluation payload for that agent’s runs (a score and a verdict, a set of rubric fields, whatever your evaluation produces), plus an optional template that renders the payload as a one-line summary. Naming lets one agent carry several kinds of evaluation at once — a summary-quality schema and a policy-compliance schema, for example — evaluated and recorded independently.

The evaluation itself happens outside Prefactor, and usually after the run: an automated eval suite, a model grading the output, or a person reviewing it. Whatever produces the judgement, it attaches the result to the instance as a quality payload for the matching schema name, through the SDK or the API. Prefactor renders the summary from that schema’s template and shows both on the instance’s Quality tab.

Quality payloads are part of the audit trail, not just a mutable field. Every time a named quality payload is set, changed, or cleared, Prefactor records the change as a span within that instance — a quality span, written by the platform itself and kept apart from the agent’s own activity. The evaluation history of a run is reconstructable from its span record, the same way the run itself is.

Eval runs and live runs

Once you evaluate runs routinely, not every run is production traffic. Each instance records a purpose: Live for a real run, Eval for an evaluation run, or Smoke test for a pipeline check. Your integration sets the purpose when the run is registered; it defaults to Live. The purpose is shown wherever the instance appears, so evaluation traffic is distinguishable from the traffic it is meant to protect.

Performance

Performance is whether the agent, as a whole, is delivering the business value it was put in place to deliver — not whether any individual output was correct, but whether the initiative is working.

Prefactor does not compute a business performance score. What it provides is a continuous run history (and the quality evaluations attached to it) against which you assess change outside the product: across many runs, over time, against the outcomes the agent was intended to drive.

Tracking change over time

Agents change for reasons that are not always obvious or deliberate. The underlying model may be updated by the provider. Code or prompts may be revised. The way people interact with the agent shifts as it becomes familiar. Any of these can affect quality and performance — and without a continuous record, it is hard to know whether something changed, when it changed, or what caused it.

Prefactor’s run history — and the quality evaluations attached to it — gives you a continuous record against which change becomes visible. When something shifts, for better or worse, you have the before and after to compare.