eval_ or insights_), and earlier attempts stay in the history.
Statuses
An attempt isqueued, running, completed, completed_with_errors, failed, or not_run. Each of its runs has a status:
For when a pending result becomes final, see Pending verdicts and Pending frictions.
obss evals preview
Check, without changing anything, which runs Evals can score, which rubrics changed, and whether the Evals model is ready.preview_evals
Arguments
Takes only the global flags.
Examples
Preview Evals:
mode has these fields:
Notes
While your organization can use Agent mode, the preview prints what Agent mode would use, for example “Agent mode: Obss’s evals agent reads the whole run and checks the agent’s claims before it scores. Slower, and costs more. Up to 56 agent reviews on Claude Opus 5.5.” For an experiment that launched in Agent mode, it also prints “This experiment launched in Agent mode. Evals use it unless you add —mode standard.”
obss evals trigger
Score an experiment’s finished runs, for the first time or again, optionally with another model, instructions, mode, or set of runs. The agent doesn’t run again.trigger_evals
The CLI doesn’t ask for confirmation. In Ask Obss, you approve each call.
Arguments
Flags
Also takes the global flags.
Examples
Score an experiment again after you fix a rubric:
Notes
Agent mode for Evals and Insights is in private beta. Email support@obss.app to enable it for your organization.
standardscores each run in one pass with your Evals model. It’s the fastest and lowest-cost mode.agenthas Obss’s evals agent read the whole run and the changed files and check the agent’s claims before it scores. It costs more and takes a few minutes per run. Use it for long or complex runs, or when a score depends on checking what the agent did.
--mode, the attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does. To repeat the latest mode, pass the preview’s mode.latestMode.
If your organization can’t use Agent mode, a request for it is refused with agent_mode_unavailable: “Agent mode isn’t available for your organization. Use Standard mode instead.” If you left out --mode and the experiment launched in Agent mode, the message is “This experiment launched Evals in Agent mode, which your organization can’t use now. Rerun with —mode standard.” If the Evals model can’t run Agent mode, the request is refused with agent_mode_model_unsupported and a setupUrl to Obss AI features, for example “Agent mode can’t use Claude Pro / Max models. Choose an Evals model from an API-key provider, or use Standard mode.”
obss evals list
List an experiment’s Evals attempts, with each one’s status, mode, and models.list_evals
Arguments
Flags
Also takes the global flags.
Examples
List an experiment’s Evals attempts:
eval_ ID, status, mode (standard or agent), and models. A failed attempt has error (code and message). An Agent mode attempt also has agentUsage once its agent has reviewed runs: the inputTokens, cacheReadTokens (the part your provider read from its prompt cache), and outputTokens that the evals agent has used so far. The terminal prints the mode after the status, as in agent mode · 2.4M input tokens on Claude Opus 5.5 (300K cached), and one line for each model, with its provider, model, and effort.
Notes
Each attempt records the models it started with, so later changes under Obss AI features don’t affect it. An attempt from before Obss recorded models shows Not recorded.
obss evals get
Page the per-run results of one Evals attempt, or read one run’s full verdict.get_evals_results
Arguments
Flags
Also takes the global flags.
Examples
Read the latest attempt:
Notes
A cursor stays on the attempt it came from, even after a newer attempt starts. A page can hold fewer rows than
--limit because each page also has a size limit. When something was left out of a run’s read, the CLI prints a note under its row.
obss evals compare
See which differences between an experiment’s treatments and agents hold up, task by task, from the Evals verdicts on the Matrix. The output starts as Comparisons does: one sentence that answers the experiment’s question, then a block for each difference with each side’s pass rate, how the tasks went, the result, and the next step. Every comparison follows.get_evals_comparisons
Arguments
Flags
Also takes the global flags.
Examples
Compare the latest Evals of a seven-task experiment with two treatments and two agents:
cmp_ ID. Then All 6 comparisons lists each comparison under its group, with its cmp_ ID. A likely difference prints, on its result line, the command that plans the runs to run it again. Notes come last.
Read the comparisons as JSON:
Each block in
differences has these fields:
Each comparison has these fields:
Notes
The sentence and the blocks come first. Then the most decisive comparisons come first: a clear difference, then a likely one, then the rest, and within a label the main comparisons before the breakdowns.
--kind and --label only filter which comparisons are listed: labels don’t change, and summary, differences, and comparisonCount stay the same.
--label is clear (Clear difference), likely (Likely difference), or unclear (No clear difference). unclear means that these runs can’t tell the two sides apart, not that the sides are the same.
- The main comparisons are checked together, and the breakdowns are checked together and more strictly, because there are more of them. So the same split can be clear for all agents together and likely for one agent. See How results are decided.
- One read lists up to 200 comparisons, and only as many as fit in 20 KiB. Page with
--cursorwhilehasMoreis true. If the comparisons changed since the first read, for example because Evals ran again, the cursor fails withinvalid_cursor, and you start again without it. - When a likely difference is run again, a
treatmentcomparison keeps its agent, or every agent for all agents together. Anagentcomparison keeps its treatment, and aninteractionkeeps both treatments and both agents. Plan the runs withobss experiments clone-plan --comparison. - A difference needs at least 6 tasks where one side did better than the other. Tasks where both did the same don’t count. With fewer than 6 tasks, no comparison can show a difference, and
notessays so. - An experiment with more than 10,000 Matrix cells fails with
experiment_too_large.
obss insights preview
Check whether the Insights model and Experiment insights model are ready for an experiment. This command changes nothing.preview_insights
Arguments
Takes only the global flags.
Examples
Preview Insights:
mode has these fields:
Notes
While your organization can use Agent mode, the preview prints what Agent mode would use, for example “Agent mode: Obss’s insights agent reads the whole run and checks each friction against what the tools returned. Slower, and costs more. Up to 56 agent reviews on Claude Opus 5.5.” For an experiment that launched in Agent mode, it also prints “This experiment launched in Agent mode. Insights use it unless you add —mode standard.”
obss insights trigger
Read an experiment’s finished runs for frictions, for the first time or again, optionally with other models, instructions, mode, or runs. Evals verdicts don’t change.trigger_insights
The CLI doesn’t ask for confirmation. In Ask Obss, you approve each call.
Arguments
Flags
Also takes the global flags.
Examples
Find frictions again on an experiment:
Notes
When Insights finish, Obss groups the findings into the project’s frictions. Read them with
obss frictions list.
Both modes use your Insights model and Experiment insights model, and report frictions in the same way. Agent mode is in private beta.
standardreviews each run in one pass with your Insights model. It’s the fastest and lowest-cost mode.agenthas Obss’s insights agent read the whole run and check each friction against what the tools returned. It costs more and takes a few minutes per run. Use it for long or complex runs, or when one pass misses frictions.
obss experiments brief). Without --mode, the attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does. To repeat the latest mode, pass the preview’s mode.latestMode.
If your organization can’t use Agent mode, a request for it is refused with agent_mode_unavailable: “Agent mode isn’t available for your organization. Use Standard mode instead.” If you left out --mode and the experiment launched in Agent mode, the message is “This experiment launched Insights in Agent mode, which your organization can’t use now. Rerun with —mode standard.” If the Insights model or the Experiment insights model can’t run Agent mode, the request is refused with agent_mode_model_unsupported and a setupUrl to Obss AI features. The message names the model, for example “Agent mode can’t use Claude Pro / Max models. Choose an Insights model from an API-key provider, or use Standard mode.”
obss insights list
List an experiment’s Insights attempts, with each one’s status, mode, and models.list_insights
Arguments
Flags
Also takes the global flags.
Examples
List an experiment’s Insights attempts:
insights_ ID, status, mode (standard or agent), and models: the Insights model and Experiment insights model it started with.
A failed attempt has error (code and message). In Agent mode, when the insights agent couldn’t group the frictions, error.message starts with “Insights could not group this experiment’s frictions.” and gives the cause and the next step.
An Agent mode attempt also has agentUsage once its agent has reviewed runs: the inputTokens, cacheReadTokens (the part your provider read from its prompt cache), and outputTokens that the insights agent has used so far. It includes grouping the frictions and the pairs of runs in the Brief (obss experiments brief).
The terminal prints the mode after the status, as in agent mode · 2.4M input tokens on Claude Opus 5.5 (300K cached), and one line for each model. The usage line names a model only when the Insights model and the Experiment insights model are the same.
obss insights get
Page the per-run results of one Insights attempt, read its report, or read one run’s full findings.get_insights_results
Arguments
Flags
Also takes the global flags.
Examples
Read the latest attempt:
Notes
A cursor stays on the attempt it came from, even after a newer attempt starts. The
attempt_report and trial_evidence views page with --offset, which doesn’t name the attempt, so pages after the first need the insights-id argument.
Next steps
Evals
What Evals read, Agent mode, and pending verdicts.
Insights
What Insights read, the modes, and pending frictions.
obss frictions
Read the frictions that Insights found.
Comparisons
Read the labels and run a likely difference again.
obss runs and results
Count pending and inconclusive runs with results analyze.
Obss AI features
Choose the Evals and Insights models.
