Quality evaluations

Attach quality evaluations to agent runs: declare the shape of your evaluation payload, mark evaluation runs as such, and submit the result after the run. The evaluation itself — an eval suite, a grading model, a human review — happens outside Prefactor; this page covers getting its output into the platform.

Three pieces are involved. One or more named quality schemas in the agent schema each declare what an evaluation payload looks like and how to summarise it. An instance purpose (live, smoke_test, or eval) tells Prefactor why a run happened, so evaluation traffic is distinguishable from production. And a quality payload for a given schema name — recorded on the instance after the run — carries the evaluation result. When a named payload changes, Prefactor records the change as a quality span inside the instance, so the evaluation history is auditable; you never write those spans yourself.

Declare quality schemas

Quality schemas are part of the agent schema, alongside the span type definitions described in Schemas and result schemas. In the SDK this is agentSchema / the agent schema object; in the platform it is the activity schema. Each quality schema has a name — the key you pass when you later record a payload against it — plus a JSON Schema for the payload and, optionally, a title, description, and a {{field}} template that Prefactor uses to render a one-line summary in the web app. Register as many named quality schemas as your agent needs: a summary-quality schema and a policy-compliance schema for the same runs, for example.

In TypeScript, add a quality_schemas array to the agentSchema object in httpConfig:

const agentSchema = {
  span_schemas: {
    // ... span type definitions ...
  },
  quality_schemas: [
    {
      name: 'summary_quality',
      schema: {
        type: 'object',
        properties: {
          overall_score: { type: 'number' },
          verdict: { type: 'string' },
          comments: { type: 'string' },
        },
        required: ['overall_score', 'verdict'],
      },
      template: 'Scored {{overall_score}}/100 ({{verdict}}): {{comments}}',
    },
  ],
};

In Python, register each one on the SchemaRegistry by name:

from prefactor_core.schema_registry import SchemaRegistry

registry = SchemaRegistry()
# ... registry.register(...) calls for span types ...
registry.register_quality_schema(
    name="summary_quality",
    schema={
        "type": "object",
        "properties": {
            "overall_score": {"type": "number"},
            "verdict": {"type": "string"},
            "comments": {"type": "string"},
        },
        "required": ["overall_score", "verdict"],
    },
    template="Scored {{overall_score}}/100 ({{verdict}}): {{comments}}",
)

Call register_quality_schema again with a different name to declare a second quality schema; each name must be unique within the agent schema.

Set the run’s purpose

Purpose is set when the instance is registered and defaults to live when omitted.

In Python, pass it when creating the instance:

handle = await client.create_agent_instance(
    agent_version={"name": "My Agent"},
    purpose="eval",
)

In TypeScript, purpose is an option on startInstance for code that drives the instance lifecycle directly through the core runtime:

import { createCore } from '@prefactor/core';

const core = createCore({
  httpConfig: {
    apiUrl: process.env.PREFACTOR_API_URL!,
    apiToken: process.env.PREFACTOR_API_TOKEN!,
    agentIdentifier: '1.0.0',
    agentSchema,
  },
});

core.agentManager.startInstance({ purpose: 'eval' });

The provider integrations (LangChain, AI SDK, and the rest) start instances themselves and do not yet accept a purpose, so runs they register default to live.

Record a quality payload

Once your evaluation has produced a result, record it against the instance under the name of the quality schema it matches. Passing None (Python) or null (TypeScript) removes the recorded payload for that name; each change is recorded as a quality span. Other names on the same instance are left unchanged.

In Python, from the instance handle or the client:

await handle.record_quality(
    name="summary_quality",
    payload={
        "overall_score": 87,
        "verdict": "pass",
        "comments": "Accurate summary; minor tone drift.",
    },
)
# or, with just the instance id:
await client.record_quality(instance_id, name="summary_quality", payload={...})

In TypeScript, code that drives the instance lifecycle directly through the core runtime (the same core from Set the run’s purpose) can call the instance manager directly:

core.agentManager.recordQuality({
  name: 'summary_quality',
  payload: {
    overall_score: 87,
    verdict: 'pass',
    comments: 'Accurate summary; minor tone drift.',
  },
});

The PrefactorClient returned by init() (the entry point most integrations use) doesn’t expose agentManager directly. Evaluators usually run as a separate process after the fact anyway, so the common path — for init()-based integrations and for evaluators without SDK access at all — is the HTTP API: POST /agent_instance/{agent_instance_id}/record_quality with name and payload, using the instance ID from prefactor.getAgentInstanceId() and a deployment-scoped API token (deployment tokens are allowed to record quality for exactly this reason). name must match a quality schema name already declared on the instance’s agent schema version, and payload must be a JSON object (or null to remove that name).

Worked example: two schemas on one agent

A ticket-summarization agent is a good candidate for more than one quality schema: one schema scores how good the summary is, a second checks it didn’t surface anything it shouldn’t have. Here’s the whole path, from declaring both schemas to seeing two sections on the instance’s Quality tab.

Register both quality schemas alongside the agent’s span types. This example uses the Python SchemaRegistry; the TypeScript agentSchema.quality_schemas array from Declare quality schemas works the same way:

from prefactor_core.schema_registry import SchemaRegistry

registry = SchemaRegistry()
registry.register(
    "summarize_ticket",
    {
        "type": "object",
        "properties": {"ticket_id": {"type": "string"}},
        "required": ["ticket_id"],
    },
)

registry.register_quality_schema(
    name="summary_quality",
    schema={
        "type": "object",
        "properties": {
            "overall_score": {"type": "number", "minimum": 0, "maximum": 100},
            "verdict": {"type": "string"},
        },
        "required": ["overall_score", "verdict"],
    },
    template="Scored {{overall_score}}/100 ({{verdict}})",
)
registry.register_quality_schema(
    name="policy_compliance",
    schema={
        "type": "object",
        "properties": {
            "compliant": {"type": "boolean"},
            "notes": {"type": "string"},
        },
        "required": ["compliant"],
    },
    template="Compliant: {{compliant}} — {{notes}}",
)

overall_score is out of 100, so its schema adds minimum and maximum to enforce that range, rather than leaving it as an unconstrained number.

Pass the registry into the client config and register the run. The agent’s own instrumentation records its spans as usual — see Schemas and result schemas for that part — so it’s omitted here:

config = PrefactorCoreConfig(
    http_config=HttpClientConfig(
        api_url="https://app.prefactorai.com",
        api_token=os.environ["PREFACTOR_API_TOKEN"],
    ),
    schema_registry=registry,
)
client = PrefactorCoreClient(config)
await client.initialize()

handle = await client.create_agent_instance(
    agent_version={"name": "Ticket summarizer"},
    purpose="eval",
)
# ... the agent runs, recording spans through its normal instrumentation ...
await handle.finish()

Once the run finishes, an evaluator — an eval suite, a grading model, or a person — scores it against both schemas and records each payload by name. This can happen in the same process, or, as here, in a separate script that only has the instance ID:

await client.record_quality(
    handle.instance_id,
    name="summary_quality",
    payload={"overall_score": 88, "verdict": "pass"},
)
await client.record_quality(
    handle.instance_id,
    name="policy_compliance",
    payload={"compliant": True, "notes": "No customer PII in the summary."},
)

Each call writes its own quality span on the instance, and the two names don’t interfere with each other. The instance’s Quality tab now shows a “Summary quality” section rendered as “Scored 88/100 (pass)”, and a separate “Policy compliance” section next to it. The same shape applies through TypeScript’s recordQuality or the HTTP API directly — see Record a quality payload above.

What you’ll see in the web app

Each named quality schema with a recorded payload gets its own section on the Agent › Instance › Quality tab, showing the rendered summary and the raw payload; the purpose is shown on the instance page and defaults to Live where your integration did not set one. A quality schema with no template shows the raw payload without a summary.

Quality and performance — what quality evaluations are and how Prefactor records them.
Schemas and result schemas — the agent schema that quality schemas live in.
Instance — purpose, quality payloads, and the run-level record.