Turn on Evals
1
Connect a model provider
Connect a model provider under Organization settings → Model providers.
2
Assign the Evals model
Assign the Evals model under Organization settings → Obss AI features.
3
Choose Evals for an experiment
Turn on Evals when you launch an experiment, or choose Run evals on an experiment that already ran.
Read the verdicts
Open a run and select Evals to see whether it passed, how many criteria it met, and which Evals model scored it. Expand a criterion to read its explanation.

Choose a mode
If your organization has Agent mode for Evals, pick a mode when you launch, or in Run evals or Rerun evals. Both use your Evals model and the same rubric.What Evals read
Your Evals model reads the rubric, the run’s recorded steps, and the changed files. If the run doesn’t fit the model’s context window, a note on the Evals tab (and in the Evals column of Compare) says what was left out:- A numbered range of steps didn’t fit. The model gets a count of those steps by kind and tool instead.
- A long step was shortened.
- Changed files were missing, previewed, shortened, or left out.
- The trace is larger than the 32 MB limit for one run file, so its steps couldn’t be read. Obss doesn’t save a trace that large, and the run keeps its outcome.
Runs processed before Obss added these notes show none, so on an older run a missing note doesn’t prove a full read.
Which models ran
Each Evals attempt keeps the model it started with, even if you change the Evals model later. Models show as provider, model, and effort, for example Evals model: OpenAI · GPT-5.4 · high, where default effort is the model’s own default.
An attempt that didn’t run shows no model. An older attempt from before Obss recorded models shows Not recorded in the attempt menu, the experiment details, and the CLI.
Pending verdicts
Evals read each run as it finishes. Rerun evals clears the verdicts it will rewrite, and the run’s Evals tab says Evals pending until the new one arrives. When Evals won’t give a verdict for an attempt, Obss says so. From the CLI and the plugin,obss evals get says whether each run’s verdict is waiting, final, or won’t come. obss results analyze counts runs still waiting separately from inconclusive runs.
Run Evals again
Rerunning Evals scores the same recorded work without running the agent again. If the rubric was wrong, fix it on the task, then rerun Evals. If the task itself changed, launch a new experiment instead. When Evals finish, Obss writes the experiment’s Brief again if it has Insights.- Dashboard
- CLI
- Plugin (MCP)
Open the experiment and choose Rerun evals (or Run evals if Evals haven’t run yet). You can change the Evals model and the mode for this Evals attempt.
From the CLI and the plugin
Next steps
Agent mode
Have Obss’s evals agent review each run in depth.
Insights
Find out where agents got stuck.
Read results
Read pass rates in the Matrix, and compare them for lift.
obss evals and insights
Every command and flag.
