---
title: Quality evaluations
editUrl: true
head: []
template: doc
sidebar:
  hidden: false
  attrs: {}
pagefind: true
draft: false
---

Attach quality evaluations to agent runs: declare the shape of your evaluation payload, mark evaluation runs as such, and submit the result after the run. The evaluation itself — an eval suite, a grading model, a human review — happens outside Prefactor; this page covers getting its output into the platform.

Three pieces are involved. One or more named quality schemas in the agent schema each declare what an evaluation payload looks like and how to summarise it. An instance purpose (`live`, `smoke_test`, or `eval`) tells Prefactor why a run happened, so evaluation traffic is distinguishable from production. And a quality payload for a given schema name — recorded on the instance after the run — carries the evaluation result. When a named payload changes, Prefactor records the change as a quality span inside the instance, so the evaluation history is auditable; you never write those spans yourself.

## Declare quality schemas

Quality schemas are part of the agent schema, alongside the span type definitions described in [Schemas and result schemas](/sdks/concepts-schemas). In the SDK this is `agentSchema` / the agent schema object; in the platform it is the [activity schema](/platform/concepts/activity-schema). Each quality schema has a name — the key you pass when you later record a payload against it — plus a JSON Schema for the payload and, optionally, a title, description, and a `{{field}}` template that Prefactor uses to render a one-line summary in the web app. Register as many named quality schemas as your agent needs: a summary-quality schema and a policy-compliance schema for the same runs, for example.

In TypeScript, add a `quality_schemas` array to the `agentSchema` object in `httpConfig`:

```typescript
const agentSchema = {
  span_schemas: {
    // ... span type definitions ...
  },
  quality_schemas: [
    {
      name: 'summary_quality',
      schema: {
        type: 'object',
        properties: {
          overall_score: { type: 'number' },
          verdict: { type: 'string' },
          comments: { type: 'string' },
        },
        required: ['overall_score', 'verdict'],
      },
      template: 'Scored {{overall_score}}/100 ({{verdict}}): {{comments}}',
    },
  ],
};
```

In Python, register each one on the `SchemaRegistry` by name:

```python
from prefactor_core.schema_registry import SchemaRegistry

registry = SchemaRegistry()
# ... registry.register(...) calls for span types ...
registry.register_quality_schema(
    name="summary_quality",
    schema={
        "type": "object",
        "properties": {
            "overall_score": {"type": "number"},
            "verdict": {"type": "string"},
            "comments": {"type": "string"},
        },
        "required": ["overall_score", "verdict"],
    },
    template="Scored {{overall_score}}/100 ({{verdict}}): {{comments}}",
)
```

Call `register_quality_schema` again with a different `name` to declare a second quality schema; each name must be unique within the agent schema.

## Set the run's purpose

Purpose is set when the instance is registered and defaults to `live` when omitted.

In Python, pass it when creating the instance:

```python
handle = await client.create_agent_instance(
    agent_version={"name": "My Agent"},
    purpose="eval",
)
```

In TypeScript, `purpose` is an option on `startInstance` for code that drives the instance lifecycle directly through the core runtime:

```typescript
import { createCore } from '@prefactor/core';

const core = createCore({
  httpConfig: {
    apiUrl: process.env.PREFACTOR_API_URL!,
    apiToken: process.env.PREFACTOR_API_TOKEN!,
    agentIdentifier: '1.0.0',
    agentSchema,
  },
});

core.agentManager.startInstance({ purpose: 'eval' });
```

The provider integrations (LangChain, AI SDK, and the rest) start instances themselves and do not yet accept a purpose, so runs they register default to `live`.

## Record a quality payload

Once your evaluation has produced a result, record it against the instance under the name of the quality schema it matches. Passing `None` (Python) or `null` (TypeScript) removes the recorded payload for that name; each change is recorded as a quality span. Other names on the same instance are left unchanged.

In Python, from the instance handle or the client:

```python
await handle.record_quality(
    name="summary_quality",
    payload={
        "overall_score": 87,
        "verdict": "pass",
        "comments": "Accurate summary; minor tone drift.",
    },
)
# or, with just the instance id:
await client.record_quality(instance_id, name="summary_quality", payload={...})
```

In TypeScript, code that drives the instance lifecycle directly through the core runtime (the same `core` from [Set the run's purpose](#set-the-runs-purpose)) can call the instance manager directly:

```typescript
core.agentManager.recordQuality({
  name: 'summary_quality',
  payload: {
    overall_score: 87,
    verdict: 'pass',
    comments: 'Accurate summary; minor tone drift.',
  },
});
```

The `PrefactorClient` returned by `init()` (the entry point most integrations use) doesn't expose `agentManager` directly. Evaluators usually run as a separate process after the fact anyway, so the common path — for `init()`-based integrations and for evaluators without SDK access at all — is the HTTP API: [POST /agent_instance/{agent_instance_id}/record_quality](/api/platform/operations/actionagentinstancerecordquality) with `name` and `payload`, using the instance ID from `prefactor.getAgentInstanceId()` and a deployment-scoped API token (deployment tokens are allowed to record quality for exactly this reason). `name` must match a quality schema name already declared on the instance's agent schema version, and `payload` must be a JSON object (or `null` to remove that name).

## Worked example: two schemas on one agent

A ticket-summarization agent is a good candidate for more than one quality schema: one schema scores how good the summary is, a second checks it didn't surface anything it shouldn't have. Here's the whole path, from declaring both schemas to seeing two sections on the instance's Quality tab.

Register both quality schemas alongside the agent's span types. This example uses the Python `SchemaRegistry`; the TypeScript `agentSchema.quality_schemas` array from [Declare quality schemas](#declare-quality-schemas) works the same way:

```python
from prefactor_core.schema_registry import SchemaRegistry

registry = SchemaRegistry()
registry.register(
    "summarize_ticket",
    {
        "type": "object",
        "properties": {"ticket_id": {"type": "string"}},
        "required": ["ticket_id"],
    },
)

registry.register_quality_schema(
    name="summary_quality",
    schema={
        "type": "object",
        "properties": {
            "overall_score": {"type": "number", "minimum": 0, "maximum": 100},
            "verdict": {"type": "string"},
        },
        "required": ["overall_score", "verdict"],
    },
    template="Scored {{overall_score}}/100 ({{verdict}})",
)
registry.register_quality_schema(
    name="policy_compliance",
    schema={
        "type": "object",
        "properties": {
            "compliant": {"type": "boolean"},
            "notes": {"type": "string"},
        },
        "required": ["compliant"],
    },
    template="Compliant: {{compliant}} — {{notes}}",
)
```

`overall_score` is out of 100, so its schema adds `minimum` and `maximum` to enforce that range, rather than leaving it as an unconstrained `number`.

Pass the registry into the client config and register the run. The agent's own instrumentation records its spans as usual — see [Schemas and result schemas](/sdks/concepts-schemas) for that part — so it's omitted here:

```python
config = PrefactorCoreConfig(
    http_config=HttpClientConfig(
        api_url="https://app.prefactorai.com",
        api_token=os.environ["PREFACTOR_API_TOKEN"],
    ),
    schema_registry=registry,
)
client = PrefactorCoreClient(config)
await client.initialize()

handle = await client.create_agent_instance(
    agent_version={"name": "Ticket summarizer"},
    purpose="eval",
)
# ... the agent runs, recording spans through its normal instrumentation ...
await handle.finish()
```

Once the run finishes, an evaluator — an eval suite, a grading model, or a person — scores it against both schemas and records each payload by name. This can happen in the same process, or, as here, in a separate script that only has the instance ID:

```python
await client.record_quality(
    handle.instance_id,
    name="summary_quality",
    payload={"overall_score": 88, "verdict": "pass"},
)
await client.record_quality(
    handle.instance_id,
    name="policy_compliance",
    payload={"compliant": True, "notes": "No customer PII in the summary."},
)
```

Each call writes its own quality span on the instance, and the two names don't interfere with each other. The instance's [Quality tab](/admin-ui/agent-instance/quality) now shows a "Summary quality" section rendered as "Scored 88/100 (pass)", and a separate "Policy compliance" section next to it. The same shape applies through TypeScript's `recordQuality` or the HTTP API directly — see [Record a quality payload](#record-a-quality-payload) above.

## What you'll see in the web app

Each named quality schema with a recorded payload gets its own section on the [Agent › Instance › Quality tab](/admin-ui/agent-instance/quality), showing the rendered summary and the raw payload; the purpose is shown on the instance page and defaults to Live where your integration did not set one. A quality schema with no template shows the raw payload without a summary.

## Related

- [Quality and performance](/platform/quality-and-performance) — what quality evaluations are and how Prefactor records them.
- [Schemas and result schemas](/sdks/concepts-schemas) — the agent schema that quality schemas live in.
- [Instance](/platform/concepts/instance) — purpose, quality payloads, and the run-level record.