# Evals - Obss

Evals and Insights
Evals
See each run’s pass or fail verdicts, choose a scoring mode, check what Evals read, and score runs again after a rubric change.
Evals
tell you whether each finished run met the task’s rubric. Your
Evals model
marks each criterion pass or fail with an explanation, and those verdicts become the pass rates on the
Matrix
.
Evals judge the outcome, while
Insights
look at how the agent worked and find
frictions
. You can turn on either, both, or neither, and rerun each one on its own.
Turn on Evals
1
Connect a model provider
Connect a
model provider
under
Organization settings → Model providers
.
2
Assign the Evals model
Assign the
Evals model
under
Organization settings →
Obss AI features
.
3
Choose Evals for an experiment
Turn on
Evals
when you
launch an experiment
, or choose
Run evals
on an experiment that already ran.
Read the verdicts
Open a run and select
Evals
to see whether it passed, how many criteria it met, and which
Evals model
scored it. Expand a criterion to read its explanation.
Because a model judges the run,
write each criterion
so that a reader could mark it true or false for one run.
Choose a mode
If your organization has Agent mode for Evals, pick a mode when you launch, or in
Run evals
or
Rerun evals
. Both use your
Evals model
and the same rubric.
Mode
How it scores
Standard
Scores each run in one pass. It’s the fastest and lowest-cost mode.
Agent
(private beta)
Obss’s evals agent reads the full run and the changed files, and checks the agent’s claims before it scores. It costs more and takes a few minutes per run.
What Evals read
Your
Evals model
reads the rubric, the run’s recorded steps, and the changed files. If the run doesn’t fit the model’s context window, a note on the
Evals
tab (and in the
Evals
column of
Compare
) says what was left out:
A numbered range of steps didn’t fit. The model gets a count of those steps by kind and tool instead.
A long step was shortened.
Changed files were missing, previewed, shortened, or left out.
The trace is larger than the 32 MB limit for one run file, so its steps couldn’t be read. Obss doesn’t save a trace that large, and the run keeps its outcome.
No note means the model read everything. A verdict covers only what the model read.
If the work still doesn’t fit after Obss shortens it, Evals fail for that run and the message names the
Evals model
. Choose a model with a larger context window under
Obss AI features
and run Evals again. A failed Evals read isn’t a friction.
Runs processed before Obss added these notes show none, so on an older run a missing note doesn’t prove a full read.
Which models ran
Each Evals attempt keeps the model it started with, even if you change the
Evals model
later. Models show as provider, model, and effort, for example
Evals model: OpenAI · GPT-5.4 · high
, where
default
effort is the model’s own default.
Where
What it shows
A run’s
Evals
tab
The
Evals model
, also while the verdict is pending and when Evals failed
Any tab of a run
The help icon next to
Evals
names the verdict’s mode and
Evals model
The info icon next to the experiment status
Latest evals and insights
: the models of the latest attempts
The attempt menu on
Matrix
Each attempt’s models, when there is more than one attempt
CLI and Plugin (MCP)
Each attempt’s models, in the Evals history and results
obss data query
and the plugin’s
Query Obss data
The
evals
dataset’s
evalsModelId
,
evalsModelName
, and
mode
, to compare results by model or mode
A public experiment link
The same models as the dashboard
An attempt that didn’t run shows no model. An older attempt from before Obss recorded models shows
Not recorded
in the attempt menu, the experiment details, and the CLI.
Pending verdicts
Evals read each run as it finishes.
Rerun evals
clears the verdicts it will rewrite, and the run’s
Evals
tab says
Evals pending
until the new one arrives. When Evals won’t give a verdict for an attempt, Obss says so.
From the CLI and the plugin,
obss evals get
says whether each run’s verdict is waiting, final, or won’t come.
obss results analyze
counts runs still waiting separately from inconclusive runs.
Run Evals again
Rerunning Evals scores the same recorded work without running the agent again. If the rubric was wrong, fix it on the task, then rerun Evals. If the task itself changed, launch a new experiment instead. When Evals finish, Obss writes the experiment’s
Brief
again if it has Insights.
Dashboard
CLI
Plugin (MCP)
Open the experiment and choose
Rerun evals
(or
Run evals
if Evals haven’t run yet). You can change the
Evals model
and the mode for this Evals attempt.
obss
evals
preview
exp_8f2c41d0
obss
evals
trigger
exp_8f2c41d0
The preview says which runs can get Evals, what changed in the rubric, and whether the provider is ready.
obss evals trigger
lists every flag.
Ask your AI assistant to run Evals again. It calls
Start Evals for an experiment
(
trigger_evals
), and you approve each call.
From the CLI and the plugin
Task
CLI
Plugin (MCP)
Preview Evals before you run them
obss evals preview
preview_evals
Run Evals
obss evals trigger
trigger_evals
List Evals attempts
obss evals list
list_evals
Read Evals results
obss evals get
get_evals_results
Compare results by model or mode
obss data query
query_data
Next steps
Agent mode
Have Obss’s evals agent review each run in depth.
Insights
Find out where agents got stuck.
Read results
Read pass rates in the Matrix, and compare them for lift.
obss evals and insights
Every command and flag.
Comparisons
Agent mode
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform