Skip to main content
Evals score each finished run against its task’s rubric, and Insights read each run’s trajectory to find frictions. Each time either runs on an experiment, Obss records an attempt (eval_ or insights_), and earlier attempts stay in the history.

Statuses

An attempt is queued, running, completed, completed_with_errors, failed, or not_run. Each of its runs has a status: For when a pending result becomes final, see Pending verdicts and Pending frictions.

obss evals preview

Check, without changing anything, which runs Evals can score, which rubrics changed, and whether the Evals model is ready.
Plugin (MCP) tool: preview_evals Arguments Takes only the global flags. Examples Preview Evals:
Result fields mode has these fields: Notes While your organization can use Agent mode, the preview prints what Agent mode would use, for example “Agent mode: Obss’s evals agent reads the whole run and checks the agent’s claims before it scores. Slower, and costs more. Up to 56 agent reviews on Claude Opus 5.5.” For an experiment that launched in Agent mode, it also prints “This experiment launched in Agent mode. Evals use it unless you add —mode standard.”

obss evals trigger

Score an experiment’s finished runs, for the first time or again, optionally with another model, instructions, mode, or set of runs. The agent doesn’t run again.
Plugin (MCP) tool: trigger_evals The CLI doesn’t ask for confirmation. In Ask Obss, you approve each call. Arguments Flags Also takes the global flags. Examples Score an experiment again after you fix a rubric:
Score only the runs whose Evals failed last time:
Score in Agent mode:
Result fields Notes
Agent mode for Evals and Insights is in private beta. Email support@obss.app to enable it for your organization.
Both modes use your Evals model and score against the same rubric.
  • standard scores each run in one pass with your Evals model. It’s the fastest and lowest-cost mode.
  • agent has Obss’s evals agent read the whole run and the changed files and check the agent’s claims before it scores. It costs more and takes a few minutes per run. Use it for long or complex runs, or when a score depends on checking what the agent did.
Without --mode, the attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does. To repeat the latest mode, pass the preview’s mode.latestMode. If your organization can’t use Agent mode, a request for it is refused with agent_mode_unavailable: “Agent mode isn’t available for your organization. Use Standard mode instead.” If you left out --mode and the experiment launched in Agent mode, the message is “This experiment launched Evals in Agent mode, which your organization can’t use now. Rerun with —mode standard.” If the Evals model can’t run Agent mode, the request is refused with agent_mode_model_unsupported and a setupUrl to Obss AI features, for example “Agent mode can’t use Claude Pro / Max models. Choose an Evals model from an API-key provider, or use Standard mode.”

obss evals list

List an experiment’s Evals attempts, with each one’s status, mode, and models.
Plugin (MCP) tool: list_evals Arguments Flags Also takes the global flags. Examples List an experiment’s Evals attempts:
Result fields Each attempt has its eval_ ID, status, mode (standard or agent), and models. A failed attempt has error (code and message). An Agent mode attempt also has agentUsage once its agent has reviewed runs: the inputTokens, cacheReadTokens (the part your provider read from its prompt cache), and outputTokens that the evals agent has used so far. The terminal prints the mode after the status, as in agent mode · 2.4M input tokens on Claude Opus 5.5 (300K cached), and one line for each model, with its provider, model, and effort. Notes Each attempt records the models it started with, so later changes under Obss AI features don’t affect it. An attempt from before Obss recorded models shows Not recorded.

obss evals get

Page the per-run results of one Evals attempt, or read one run’s full verdict.
Plugin (MCP) tool: get_evals_results Arguments Flags Also takes the global flags. Examples Read the latest attempt:
Read the next page of the same attempt:
Read the next page of one run’s verdict:
Result fields Notes A cursor stays on the attempt it came from, even after a newer attempt starts. A page can hold fewer rows than --limit because each page also has a size limit. When something was left out of a run’s read, the CLI prints a note under its row.

obss evals compare

See which differences between an experiment’s treatments and agents hold up, task by task, from the Evals verdicts on the Matrix. The output starts as Comparisons does: one sentence that answers the experiment’s question, then a block for each difference with each side’s pass rate, how the tasks went, the result, and the next step. Every comparison follows.
Plugin (MCP) tool: get_evals_comparisons Arguments Flags Also takes the global flags. Examples Compare the latest Evals of a seven-task experiment with two treatments and two agents:
The sentence comes first, then the block for the one difference, with its cmp_ ID. Then All 6 comparisons lists each comparison under its group, with its cmp_ ID. A likely difference prints, on its result line, the command that plans the runs to run it again. Notes come last. Read the comparisons as JSON:
Result fields Each block in differences has these fields: Each comparison has these fields: Notes The sentence and the blocks come first. Then the most decisive comparisons come first: a clear difference, then a likely one, then the rest, and within a label the main comparisons before the breakdowns. --kind and --label only filter which comparisons are listed: labels don’t change, and summary, differences, and comparisonCount stay the same. --label is clear (Clear difference), likely (Likely difference), or unclear (No clear difference). unclear means that these runs can’t tell the two sides apart, not that the sides are the same.
  • The main comparisons are checked together, and the breakdowns are checked together and more strictly, because there are more of them. So the same split can be clear for all agents together and likely for one agent. See How results are decided.
  • One read lists up to 200 comparisons, and only as many as fit in 20 KiB. Page with --cursor while hasMore is true. If the comparisons changed since the first read, for example because Evals ran again, the cursor fails with invalid_cursor, and you start again without it.
  • When a likely difference is run again, a treatment comparison keeps its agent, or every agent for all agents together. An agent comparison keeps its treatment, and an interaction keeps both treatments and both agents. Plan the runs with obss experiments clone-plan --comparison.
  • A difference needs at least 6 tasks where one side did better than the other. Tasks where both did the same don’t count. With fewer than 6 tasks, no comparison can show a difference, and notes says so.
  • An experiment with more than 10,000 Matrix cells fails with experiment_too_large.

obss insights preview

Check whether the Insights model and Experiment insights model are ready for an experiment. This command changes nothing.
Plugin (MCP) tool: preview_insights Arguments Takes only the global flags. Examples Preview Insights:
Result fields mode has these fields: Notes While your organization can use Agent mode, the preview prints what Agent mode would use, for example “Agent mode: Obss’s insights agent reads the whole run and checks each friction against what the tools returned. Slower, and costs more. Up to 56 agent reviews on Claude Opus 5.5.” For an experiment that launched in Agent mode, it also prints “This experiment launched in Agent mode. Insights use it unless you add —mode standard.”

obss insights trigger

Read an experiment’s finished runs for frictions, for the first time or again, optionally with other models, instructions, mode, or runs. Evals verdicts don’t change.
Plugin (MCP) tool: trigger_insights The CLI doesn’t ask for confirmation. In Ask Obss, you approve each call. Arguments Flags Also takes the global flags. Examples Find frictions again on an experiment:
Review in Agent mode:
Result fields Notes When Insights finish, Obss groups the findings into the project’s frictions. Read them with obss frictions list. Both modes use your Insights model and Experiment insights model, and report frictions in the same way. Agent mode is in private beta.
  • standard reviews each run in one pass with your Insights model. It’s the fastest and lowest-cost mode.
  • agent has Obss’s insights agent read the whole run and check each friction against what the tools returned. It costs more and takes a few minutes per run. Use it for long or complex runs, or when one pass misses frictions.
In Agent mode, the insights agent also does the experiment’s grouping of frictions, which Standard mode does in one pass. For each task of each difference in the Brief, it says what a failed run and a passed run did differently (obss experiments brief). Without --mode, the attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does. To repeat the latest mode, pass the preview’s mode.latestMode. If your organization can’t use Agent mode, a request for it is refused with agent_mode_unavailable: “Agent mode isn’t available for your organization. Use Standard mode instead.” If you left out --mode and the experiment launched in Agent mode, the message is “This experiment launched Insights in Agent mode, which your organization can’t use now. Rerun with —mode standard.” If the Insights model or the Experiment insights model can’t run Agent mode, the request is refused with agent_mode_model_unsupported and a setupUrl to Obss AI features. The message names the model, for example “Agent mode can’t use Claude Pro / Max models. Choose an Insights model from an API-key provider, or use Standard mode.”

obss insights list

List an experiment’s Insights attempts, with each one’s status, mode, and models.
Plugin (MCP) tool: list_insights Arguments Flags Also takes the global flags. Examples List an experiment’s Insights attempts:
Result fields Each attempt has its insights_ ID, status, mode (standard or agent), and models: the Insights model and Experiment insights model it started with. A failed attempt has error (code and message). In Agent mode, when the insights agent couldn’t group the frictions, error.message starts with “Insights could not group this experiment’s frictions.” and gives the cause and the next step. An Agent mode attempt also has agentUsage once its agent has reviewed runs: the inputTokens, cacheReadTokens (the part your provider read from its prompt cache), and outputTokens that the insights agent has used so far. It includes grouping the frictions and the pairs of runs in the Brief (obss experiments brief). The terminal prints the mode after the status, as in agent mode · 2.4M input tokens on Claude Opus 5.5 (300K cached), and one line for each model. The usage line names a model only when the Insights model and the Experiment insights model are the same.

obss insights get

Page the per-run results of one Insights attempt, read its report, or read one run’s full findings.
Plugin (MCP) tool: get_insights_results Arguments Flags Also takes the global flags. Examples Read the latest attempt:
Read the next page of an attempt report:
Result fields Notes A cursor stays on the attempt it came from, even after a newer attempt starts. The attempt_report and trial_evidence views page with --offset, which doesn’t name the attempt, so pages after the first need the insights-id argument.

Next steps