LLM provider

An LLM provider is a connection from your account to a language model service, together with the API key Prefactor uses to call it.

Hosted judges run on language models, and Prefactor calls those models with credentials you supply. A provider holds the connection once. Each model under it is a specific model on that service, and it is the model you pick when you configure a hosted judge. Judges are the only thing that uses providers today.

Prefactor supports Anthropic, OpenAI, Ollama Cloud, and Together AI, plus a custom OpenAI-compatible endpoint for any other service that speaks the OpenAI API. For the first four, the endpoint is already known, so a provider is a name and an API key. A custom endpoint also takes its URL, which must be a public HTTPS address. A provider’s type is fixed when you create it.

For the four named providers, each model must be one that the models.dev catalogue lists for that provider. You pick it from a list or type its API name, and Prefactor fills in the model’s display name and the identifier it uses to look up pricing. For a custom endpoint you name the model yourself and give the model name the endpoint expects.

Keys and credentials

Prefactor encrypts the API key at rest. Once you save it, neither the web app nor the API shows it again, and editing a provider without entering a new key leaves the stored key in place. Prefactor does not test the key when you save the provider, so a wrong key surfaces the first time a hosted judge calls the model, and that judge run fails.

How providers fit

Providers belong to the account, not to an environment or an agent. Every hosted judge in the account can use any model from any of its providers. A hosted judge version records the model it was created with, and versions do not change after creation, so switching a judge to a different model means creating a new version.

Deleting a provider deletes its models too. Prefactor refuses to delete a model, or a provider with such a model, while any hosted version refers to it.

In the Admin UI

• Account › LLM providers tab — create, edit, and delete providers and their models.
• Judge page › Configuration tab — where a hosted judge version picks its model.

From the CLI

• Set up a judge from the CLI — prefactor llm_providers and prefactor llm_provider_models, then a judge version on the model.

Related concepts

• Account — providers and their keys belong to the account.
• Agent — a hosted judge is an agent whose versions name a provider model.