# Comparisons - Obss

Read results
Comparisons
See who did better, how sure the result is, and what to do next.
Agents don’t pass the same task every time, so a small gap in pass rate may not hold up. Open an experiment and select the
Comparisons
tab to see which differences do. It compares
Evals
verdicts, so it needs Evals.
A difference needs at least 6 tasks where one side did better than the other. With fewer tasks, no difference can show, however many runs you add.
Answers
Comparisons opens with one sentence that answers your experiment’s question, and then one block for each difference:
The sentence covers your experiment’s
main comparisons
: each treatment against the
baseline
, for all agents together. With one agent, they’re that agent’s comparisons. With one treatment, they’re each pair of agents. For example:
“Stripe CLI did better than Raw Agent on all 7 tasks.”
“Stripe MCP likely did better than Raw Agent on 6 tasks, same on 4.” A likely result is never stated as settled.
“No treatment did clearly better than All official CLIs across all agents.”
A line under the sentence counts the likely results for one agent alone or between agents, for example “7 other comparisons show a likely difference. See All 40 comparisons.”
Each difference then gets a block of its own, and every block has the same parts: the two sides with their pass rates, how the tasks went (“Better on 14 tasks, worse on 2, same on 26.”), the result and its next step, and how each agent did, with
See each agent
to open those rows in
All comparisons
. A main comparison gets a block when its result is a
Clear difference
or a
Likely difference
. A result for one agent alone or between agents gets one only when it’s a
Clear difference
. A comparison with no clear difference gets no block: the sentence says it, and
All comparisons
lists it. Up to three blocks show, and
Show all N differences
opens the rest.
Comparisons read the latest Evals attempt. To read an earlier one, choose it in the attempt menu, as on the
Matrix
. The
Matrix
also shows the sentence in one line above its grid.
Brief
When Insights has reviewed the runs, each difference’s block gets a
Brief
, up to 5 per experiment. Its sentence says what the side that did worse ran into, for example “Raw Agent runs had no authenticated Stripe access.” Under it, “Frictions on the 7 tasks where Raw Agent did worse” lists up to 3 frictions from that side’s runs, with how many of that side’s failed runs and the other side’s passed runs had each one (“Raw Agent 24 of 28 failed runs”, “Stripe CLI 0 of 26 passed runs”). Select a friction to open its suggested fix.
A friction under a result was seen with the side that did worse. That doesn’t mean it caused the difference.
Copy as issue:
a block with a Brief has a
⋯
menu.
Copy as issue
copies the difference as Markdown for GitHub or another tracker: its sides, result, sentence, frictions, and links, and in Agent mode the rows of
What the runs did differently
that the block shows first.
After a fix:
when a
Clear difference
lists a friction to fix in your product,
After a fix, run the N tasks that differed again · N runs
under the list opens a new experiment with those runs, so you can check your fix.
What the runs did differently
:
with Insights in Agent mode (private beta), a block with a Brief ends with pairs of runs: on each task where that side did worse, one of its failed runs next to a passed run of the other side, and the step where they differ. Once every pair is read, the sentence says what the runs did across the tasks, and the section shows one example task and any task that went differently.
When nothing differs, there’s no Brief. The experiment’s frictions are on its
Frictions
tab.
What you see
What to do
No Brief
If Insights hasn’t reviewed the runs,
run Insights
. If the experiment, its Evals, or its Insights is still running, wait.
Brief
and “Being written.”
Wait. The sentence appears on its own.
”The Brief could not be written.”
The frictions still show. If the line adds “Check the Experiment insights model in Obss AI features”, check that the model is assigned and that the person who started the Evals or Insights can use its provider. Otherwise, the Brief is written again the next time Evals or Insights finish.
Frictions with no
Brief
label
The Brief was written for earlier results, or the experiment finished before Obss added the Brief. It’s written the next time Evals or Insights finish.
How the Brief is made
Which frictions:
a friction is listed when it’s in at least 3 runs of the side that did worse, on at least 2 tasks, and in a larger share of that side’s runs than the other side’s. The largest gap comes first. Runs that Insights didn’t review are left out, and a line under the list says how many. Frictions caused by an Obss error are left out.
Numbers:
Obss picks the differences and counts the runs. No model picks or counts anything.
The sentence:
your organization’s
Experiment insights model
writes it, with the model access of the person who started the Evals or Insights, and without your custom instructions. See
Obss AI features
. Obss rejects a sentence with a number or a claim of cause, and asks again. With Insights in
Agent mode
, once the insights agent has read every pair of runs of a difference, the sentence also reads their lines and says what the runs did across the tasks.
Updates:
Obss writes the Brief when the experiment, its Evals, and its Insights finish, and again each time Evals or Insights finishes. In Agent mode, it also writes the Brief again once, when the last pair of runs is read. Until a new sentence is ready, the old one is hidden. Only the latest Evals attempt shows a Brief.
Result labels
The result says how sure the answer is:
Result
What it means
What to do
Clear difference
One side did better on enough tasks that the gap holds up.
Act on it.
Likely difference · Run again to be sure · N runs
One side is ahead, but more runs are needed to be sure.
Select
Run again to be sure
. See
Run a comparison again
.
Likely difference · Needs more tasks to be sure
One side is ahead, but running the same tasks again can’t settle it.
Add tasks, and launch again.
No clear difference
Neither side is reliably ahead on these tasks. That doesn’t mean they’re the same.
Add tasks if the difference matters to you.
No clear difference
, with too few tasks
The sentence at the top says the tasks are too few, for example “4 tasks are too few to compare Stripe CLI and Raw Agent.”
Add tasks, and launch again.
Point to a result to see what it means and its p-values.
How results are decided
explains the rules.
All comparisons
Under the blocks,
All N comparisons
lists every comparison in one table:
Column
What it shows
Compared
The two sides, each on its own line
Runs passed
Each side’s pass rate, and its passing runs out of the runs compared
Task by task
Which side did better on how many tasks, for example “Stripe CLI better on 6, same on 1”
Result
The result and its next step, on one line, as in a block
The rows are in groups:
Group
What it compares
Treatments across all agents
Each treatment against the baseline, or each pair of treatments with no baseline. With one agent, the group is
Treatments
.
Treatments with
an agent
The same, for one agent
Agents with
a treatment
Each pair of agents with one treatment. With one treatment, the group is
Agents
.
Does a treatment help one agent more?
The last group. Each clear or likely case is a sentence, for example “Stripe CLI helped Codex more than Claude Code.” With none, it says “No treatment clearly helped one agent more than another.”
With more than 6 agents, Obss doesn’t compare agents with each other.
Run a comparison again to be sure
You can run a
Likely difference
again: Obss plans a new experiment with the fewest runs needed to settle it. Launching it uses runs from your run balance.
Run again to be sure · N runs
appears once the experiment and its Evals finish, on the latest Evals attempt. A
Likely difference
shows
Needs more tasks to be sure
instead when fewer than 6 of its tasks can still show a difference, or when settling it would take more than 10 runs of each task.
Dashboard
CLI
Plugin (MCP)
On the block’s result line, or on its row in
All N comparisons
, select
Run again to be sure · N runs
.
New experiment
opens with those runs selected. Check the run count in the footer, then select
Launch
.
If the comparison can no longer be run again, for example because Evals ran again or a treatment was deleted,
Comparisons
reopens with a note that explains why.
List the likely differences. Each one prints the command that plans its runs:
obss
evals
compare
exp_8f2c41d0
--label
likely
Save the plan, check it, and launch it:
obss
experiments
clone-plan
exp_8f2c41d0
\
--comparison
cmp_4a0e2c9d
--output
json
\
|
jq
'.data.plan'
>
run-again.json
obss
experiments
preflight
--plan-file
./run-again.json
obss
experiments
launch
--plan-file
./run-again.json
obss evals compare
and
obss experiments clone-plan
list every flag and field.
Ask your AI assistant, for example:
Did MCP + Skills do better than Raw Agent in my last experiment? If the result is likely, plan the runs that check it again and tell me the run count.
It calls
Compare treatments and agents
(
get_evals_comparisons
), then
Get an experiment’s launch plan
(
get_experiment_plan
) with the
comparison_id
, then
Check an experiment plan before launch
(
preflight_experiment
) and
Launch an experiment
(
launch_experiment
). A launch uses runs, so turn on tool approval in your AI assistant.
When the new experiment finishes, open its
Comparisons
. It has only one main comparison, so if the result holds, it’s a
Clear difference
. Whether a treatment helps one agent more is the exception: in the new experiment it’s still a breakdown, so it can come out likely again.
What the new experiment includes
Tasks:
every task from the comparison that can still show a difference, with at least 6. A task is left out when it already ran at least 4 times on both sides together, and every run passed or every run failed.
Sides:
the comparison’s treatments and agents. For all agents together, every agent.
Runs each:
the fewest that are likely to settle the difference, from the experiment’s own (at least 3) up to 10.
Versions:
the experiment’s task versions, and each treatment’s current tools.
Evals:
on, with the mode, Evals model, and instructions of the Evals attempt that found the difference.
Name:
“Run again:”, followed by the comparison.
Empty states
When there’s nothing to compare, the tab shows one of these messages:
Message
What to do
Nothing to compare
Launch again with a second treatment or agent.
Too many agents to compare
The experiment ran one treatment with more than 6 agents. Launch again with fewer.
Evals off
Run Evals
.
No evaluated runs yet
Wait for Evals to finish.
No evaluated runs to compare
No comparison has Evals verdicts on both sides. Run Evals again.
Too large to compare
The experiment has more runs than Comparisons can read at once. Read its pass rates on the
Matrix
.
Public experiment links
don’t show
Comparisons
.
How results are decided
On the dashboard, open
How results are decided
under the table. It sums up the rules below and lists what applies to your experiment:
Obss compares tasks, not runs. For each comparison, it counts the tasks where one side passed more often than the other, and ignores the tasks where both did the same. An exact two-sided sign test gives the p-value: the probability of a split at least this uneven if neither side were really better. Five tasks that all go one way give p = 0.063, and six give p = 0.031, which is why a difference needs at least 6 tasks.
To keep each task fair, both sides use the same number of runs with a verdict, and every agent counts equally. So
Runs passed
can be lower than the run count on the
Matrix
.
Main comparisons and breakdowns
The more comparisons an experiment makes, the more likely one of them shows a large gap that isn’t real. So Obss checks the comparisons in two groups, and applies the Holm correction within each group:
Group
What it holds
Main comparisons
The comparisons that answer the experiment’s question. See
Answers
.
Breakdowns
Every other comparison: each agent on its own, agents against each other when treatments vary, and whether a treatment helps one agent more. There are more of them, so they’re checked more strictly.
Clear difference:
p-value below 0.05 after the correction within its group.
Likely difference:
below 0.05 before the correction, but not after.
No clear difference:
everything else.
For example, an experiment with 2 treatments, 2 agents, and 7 tasks has 1 main comparison and 5 breakdowns. Stripe CLI did better on all 7 tasks, for both agents: p = 0.016. For all agents together, that’s a
Clear difference
. For each agent on its own, it’s a
Likely difference
, because it’s checked together with 4 other breakdowns.
How results are decided
then says “With 7 tasks, a result for one agent alone or between agents can be a likely difference at most.”
When there’s only one main comparison, nothing corrects it, so it’s never a
Likely difference
: it’s clear or not. A breakdown can be clear when the main comparison isn’t, for example when a treatment helped one agent and hurt another.
How results are decided
also lists what limits these results, such as tasks that passed in every run, runs with no Evals verdict, or how often repeated runs disagreed. While the experiment or its Evals is still running, a line at the top of the page says the results can still change.
From the CLI and the plugin
Task
CLI
Plugin (MCP)
Compare treatments and agents
obss evals compare
get_evals_comparisons
Read the differences with their Briefs
obss experiments brief
get_experiment_brief
Plan the runs that check a difference again
obss experiments clone-plan
get_experiment_plan
Each command gives the sentence first, then each difference, in the dashboard’s words. Its reference page lists every field.
Next steps
Find and fix frictions
Change one thing, and launch again to test it.
Measure a tool's impact
Check that a tool’s lift is real.
Compare agents and models
Choose an agent on a difference that holds up.
Frictions
See every friction and its suggested fix.
Matrix and runs
Evals
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform