Skip to content
PrefactorPrefactorPrefactor Docs

Record a quality payload on an agent instance

POST
/api/v1/agent_instance/{agent_instance_id}/record_quality
curl --request POST \
  --url https://app.prefactorai.com/api/v1/agent_instance/013xrzp12g3nqk8ns6jadrqmpgrjkeny/record_quality \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '"summary_quality"'

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
agent_instance_id
required
AgentInstance_Id

A valid agent instance ID

string
Example
013xrzp12g3nqk8ns6jadrqmpgrjkeny

Request Body

Section titled “Request Body”

Quality schema name (key in the agent schema version quality_schemas)

string
Example
summary_quality

Responses

Section titled “Responses”

200

Section titled “200”

Success

Media typeapplication/json
object
details
AgentInstance_Details
object
account_id
Account_Id

A valid account ID

string
agent_deployment_id
AgentDeployment_Id

A valid agent deployment ID

string
agent_id
Agent_Id

A valid agent ID

string
agent_version_id
AgentVersion_Id

A valid agent version ID

string
cost_breakdown
One of:
AgentInstance_CostBreakdown
object
cache_read

Total cache-read tokens across all models

integer
cache_read_cost
One of:

Estimated total USD cost for cache-read tokens

number
groups

Per-model breakdown

Array<object>
AgentInstance_CostBreakdown_Group
object
cache_read

Total cache-read tokens for this model

integer
cache_read_cost
One of:

Estimated USD cost for cache-read tokens

number
context_limit
One of:

Context window size from models.dev pricing, when known

integer
input

Total input tokens for this model

integer
input_cost
One of:

Estimated USD cost for input tokens

number
model_name

Model identifier from span payloads

string
output

Total output tokens for this model

integer
output_cost
One of:

Estimated USD cost for output tokens

number
provider
One of:

Provider identifier from span payloads

string
total_cost
One of:

Estimated total USD cost for this model

number
input

Total input tokens across all models

integer
input_cost
One of:

Estimated total USD cost for input tokens

number
output

Total output tokens across all models

integer
output_cost
One of:

Estimated total USD cost for output tokens

number
total_cost
One of:

Estimated total USD cost

number
environment_id
Environment_Id

A valid environment ID

string
finished_at
One of:

When the agent instance finished (null if still in progress)

string format: date-time
id
AgentInstance_Id

A valid agent instance ID

string
inserted_at

When the agent instance was created

string format: date-time
last_activity_span_at
One of:
string format: date-time
purpose

Why this instance ran: ‘live’ for an actual agent run, ‘smoke_test’ for a pipeline check, or ‘eval’ for an evaluation run

string
Allowed values: live smoke_test eval
quality_payloads

Map of quality schema name to evaluation payload for this instance

object
key
additional properties
object
key
additional properties
any
quality_summaries

Map of quality schema name to rendered summary (only names whose template renders)

object
key
additional properties

Human-readable quality summary rendered from the quality schema template using the matching quality payload

string
raised_alert_count
One of:
integer
risk_score
One of:
AgentInstance_RiskScore
object
all_data_categories

Unique included data categories across params and results of span types that ran, in canonical order

Array<string>
Allowed values: personal_identifiers contact_information financial_information health_and_medical criminal_justice authentication_and_secrets organisational_confidential minors_data location_and_tracking behavioural_and_inferred gdpr_racial_or_ethnic_origin gdpr_political_opinions gdpr_religious_or_philosophical_beliefs gdpr_trade_union_membership gdpr_genetic_data gdpr_biometric_for_identification gdpr_sex_life_or_sexual_orientation
allowed_actions

Actions allowed by at least one span type that ran in this instance, in canonical order

Array<string>
Allowed values: create_data read_data update_data destroy_data financial_transactions external_communication
any_assessable

Whether the instance has at least one span type with scoreable data_risk

boolean
critical_threshold

Critical threshold from the agent’s risk profile ruleset

number
peak_classification
One of:

Peak observed data risk classification across all spans in this instance. null if no spans have executed.

string
Allowed values: unknown public internal confidential restricted secret
per_type

Per-schema-type score breakdown

Array<object>
AgentInstance_RiskScore_PerType
object
count

Number of span invocations for this schema type

integer
risk_level

Risk level bucket for this schema type’s per-invocation score

string
Allowed values: low medium high critical
schema_name

Span type schema name

string
total_contribution

Type_score multiplied by count

number
type_score

Risk score for a single invocation of this schema type

number
risk_level
One of:

Overall risk level bucket for this instance, or null when not assessable

string
Allowed values: low medium high critical
total_score

Total risk score across all span types for this instance

number
span_counts
AgentSpan_SpanCounts
object
active

Number of spans with status active (running)

integer
cancelled

Number of spans with status cancelled

integer
complete

Number of spans with status complete

integer
failed

Number of spans with status failed

integer
finished

Number of spans in a finished state (complete + failed + cancelled)

integer
pending

Number of spans with status pending

integer
total

Total number of agent spans

integer
span_schema_counts
One of:
AgentSpan_SpanSchemaCounts
object
by_name

Invocation count keyed by span schema name

object
key
additional properties
integer
started_at
One of:

When the agent instance started (null if not started yet)

string format: date-time
status

Status of the agent instance: ‘pending’ (not started yet), ‘active’ (started but not finished), ‘complete’ (finished successfully), ‘failed’ (ended with failure), ‘cancelled’ (never started, cancelled), or ‘terminated’ (externally terminated)

string
Allowed values: pending active complete failed cancelled terminated
termination_reason
One of:

Reason for termination (only present when status is ‘terminated’)

string
type

Type of resource (always ‘agent_instance’)

string
Allowed values: agent_instance
updated_at

When the agent instance was last updated

string format: date-time
status
string
Allowed values: success
Example
{
  "details": {
    "account_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "agent_deployment_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "agent_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "agent_version_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "cost_breakdown": null,
    "environment_id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "finished_at": "2024-01-01T00:00:00Z",
    "id": "013xrzp12g3nqk8ns6jadrqmpgrjkeny",
    "inserted_at": "2024-01-01T00:00:00Z",
    "last_activity_span_at": "2024-01-01T00:00:00Z",
    "purpose": "live",
    "raised_alert_count": null,
    "risk_score": null,
    "span_counts": {
      "active": 2,
      "cancelled": 0,
      "complete": 5,
      "failed": 0,
      "finished": 5,
      "pending": 0,
      "total": 10
    },
    "span_schema_counts": null,
    "started_at": "2024-01-01T00:00:00Z",
    "status": "pending",
    "termination_reason": "User requested termination",
    "type": "agent_instance",
    "updated_at": "2024-01-01T00:00:00Z"
  },
  "status": "success"
}

default

Section titled “default”

Error

Media typeapplication/json
One of:
object
code
string
Allowed values: bad_request bad_authtoken not_authenticated not_permitted not_implemented unknown not_found unexpected_ref_type unexpected invalid_action alert_unknown alert_already_cleared invalid_value required_value idempotency_key_already_used
message
string
status
string
Allowed values: error
Example
{
  "code": "bad_request",
  "status": "error"
}