# Obss AI features - Obss

Set up
Obss AI features
Choose the models that score runs, find frictions, group them and write the Brief, and power Ask Obss, and add custom instructions.
The
Obss AI features
score and explain your results, separately from the agents that do the work. Each of the four has its own model. Choosing them is step 5 of the
setup checklist
.
Model
What it does
Where you see the result
Evals model
Scores each run against its task rubric, criterion by criterion.
A run’s
Evals
tab, and pass rates in the
Matrix
Insights model
Finds the frictions and patterns in each run.
The experiment’s
Frictions
tab
Experiment insights model
Groups every run’s problems into the project’s frictions, and writes the experiment’s
Brief
.
The experiment’s
Frictions
tab, and the Brief on its
Comparisons
tab
Ask Obss model
Answers questions about this project’s experiments and runs.
Ask Obss, in the dashboard and in
Slack
The models come from your
model providers
: an organization provider, or a supported personal integration such as
OpenAI (ChatGPT Subscription)
,
Anthropic (Claude Subscription)
, or
Nous Portal
. A personal integration uses the account of the person who starts the work.
Choose the models
The page shows
Default model
first, and a model for each feature under
Advanced
.
Dashboard
CLI
Plugin (MCP)
Connect a provider under
Organization settings → Model providers
, or a personal integration under
Profile → Integrations
.
Open
Organization settings → Obss AI features
.
Choose a
Default model
and an effort level. Every feature uses it unless you set its own.
Optional: under
Advanced
, choose a model for a feature.
Default model
then shows
Custom per feature
.
Select
Save changes
.
Read each feature’s model and whether it can run:
obss
project
ai
inspect
--view
roles
Then send
set_roles
with the revision from that output. Each binding names a
role
, a provider connection, a model, and an effort level:
obss
project
ai
apply
--operation-file
roles.json
Learn more in
obss project ai apply
.
Ask your AI assistant to set the feature models. It calls
Get built-in and custom agents, models, and Obss AI settings
(
inspect_project_ai
), then
Create custom agents and change agents, models, and Obss AI settings
(
manage_project_ai
).
Use the same model for every Obss AI feature, and a faster model for Ask Obss.
In the CLI and the plugin, the roles are
judge
(Evals model),
sweep
(Insights model),
consolidation
(Experiment insights model), and
ask
(Ask Obss model).
Until a provider can supply a model, a feature shows
Connect models
instead of a picker. When your organization connects its first working provider, Obss can choose a starting model for each feature that has none. It never changes a model you saved.
To use other models or custom instructions for one experiment,
override them when you launch, run, or rerun
Evals or Insights.
When a saved model cannot run
A saved model stays saved when it can’t run, for example because the provider stopped listing it or your sign-in expired. The page shows a warning, such as “Evals can’t run with its saved model”, and each feature says why. Saving the page doesn’t change features you didn’t edit. Each feature works again once you fix its cause.
Evals model
The
Evals model
marks each rubric criterion
Pass
or
Fail
and explains why. There’s no separate deterministic check. Turn Evals on when you
launch an experiment
, or select
Run evals
later on the recorded work. After the first pass, the button reads
Rerun evals
.
Mode
How it scores
Trade-off
Standard
One pass per run with your Evals model. A run longer than the context window is condensed to fit.
Fastest and lowest cost
Agent
(private beta)
Obss’s evals agent reads the full run and changed files, and verifies the agent’s claims before it scores.
Many Evals model calls, so more cost and a few minutes per run
Agent mode for Evals and Insights is in private beta. Email
support@obss.app
to enable it for your organization.
Agent mode needs an Evals model that can call tools and has a known context window. If yours can’t, choose another Evals model or use Standard mode. Agent mode can’t use Claude Pro or Max models from
Anthropic (Claude Subscription)
, so with those, choose an Evals model from an API-key provider.
Insights models
The
Insights model
reads each run’s trajectory and finds
frictions
. The
Experiment insights model
groups those findings into the project’s frictions. It also writes the experiment’s
Brief
, for the person who started the Evals or Insights that just finished, without your custom instructions. Turn Insights on at launch, or select
Run insights
later. After the first pass, the button reads
Rerun Insights
.
Insights also have a Standard mode and an
Agent
mode (private beta), in which Obss’s insights agent makes many calls to both models. Agent mode for Insights needs an Insights model and an Experiment insights model that can call tools and have a known context window, and it can’t use Claude Pro / Max models. See
Choose models for Agent mode
.
For what each feature reads, and what happens when a run doesn’t fit the context window, see
What Evals read
and
What Insights read
.
Ask Obss model
Ask Obss
is the in-app assistant. It answers questions about your experiments and runs, explains the
Matrix
and
Frictions
, drafts small plans, and finds dashboard pages. The
Ask Obss model
powers it in the dashboard and in
Slack
. It needs:
A
supported provider
.
A model with a known context window. If your provider doesn’t state one,
set it
.
Add custom instructions
Custom instructions add your organization’s guidance to the Evals, Insights, and Experiment insights models (not Ask Obss). For example, tell the Evals model to prioritize correctness over style, or the Insights model to group findings by root cause. The
Brief
doesn’t use them.
Open
Organization settings → Obss AI features
and open
Advanced
.
Under
Evals model
,
Insights model
, or
Experiment insights model
, open
Custom instructions
.
Enter the instructions, up to 16,000 characters, and select
Save changes
.
If the field is empty, Obss uses its built-in guidance. Instructions don’t change the response format or what the model reads. From the CLI or the plugin, set
customInstructions
on a role binding in
set_roles
.
From the CLI and the plugin
Task
CLI
Plugin (MCP)
Read each feature’s model and whether it can run
obss project ai inspect --view roles
inspect_project_ai
Set feature models and custom instructions
obss project ai apply
with
set_roles
manage_project_ai
Run or rerun Evals
obss evals trigger
trigger_evals
Run or rerun Insights
obss insights trigger
trigger_insights
Next steps
Runs and billing
Step 6: check your run balance.
Evals
Learn how Evals score runs.
Insights
Learn how Insights find frictions.
Ask Obss
Use the in-app assistant.
Custom agents
Billing
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform