# Compare agents and models - Obss

Use cases
Compare agents and models
Run the same tasks across agents, models, and effort levels, and read pass rate next to cost, tokens, and duration to pick the best fit.
Put every agent and model that you want to compare in one experiment, so that they share the same tasks, tools, and machines. Then use frictions to see why one does better than another.
Prerequisites
The agents to compare, turned on under
Manage → Agents
, each with a
model provider
that it supports (
Agents
lists them). Connect providers under
Organization settings → Model providers
, or under
Profile → Integrations
for agents that support a personal login.
Tasks
that stand for your real work.
Runs in your balance. Each combination of task, treatment, agent, and trial uses one run, and your providers bill the model inference (
Runs and billing
).
Steps
1
Choose the tasks
Pick tasks of different sizes that cover the work you care about. Pick more than 6:
Comparisons
needs at least 6 tasks where one agent did better, and tied tasks don’t count.
2
Use one treatment for every agent
Choose
Raw Agent
to compare the agents as their vendors ship them, or your own
treatment
to compare them with your product’s tools.
3
Choose agents, models, and effort
Select
New experiment
. Under
Agents
, select each agent, its
Model
, and its effort level.
Default
effort sends no level, so the agent uses its own default for that model. Set
Trials
to 2 or more, then
launch
. You get a
notification
when it finishes.
4
Compare pass rate with cost
Open the experiment’s
Matrix
with the
Evals
metric to see pass rates, then switch to
Cost
,
Tokens
,
Duration
,
Steps
, or
Tool calls
to see what each result costs.
5
Check which differences hold up
Open the
Comparisons
tab. With one treatment, the sentence at the top answers for every pair of agents, and each pair that differs gets a block with each agent’s pass rate and the result. With more than one treatment, every agent pair is under
All N comparisons
, in the
Agents with
groups, and a clear difference between agents also gets a block at the top. For a
Likely difference
, select
Run again to be sure · N runs
before you choose. When Insights ran, the
Brief
in a block lists the frictions seen with the agent that did worse.
6
Find out why agents differ
On the
Frictions
tab,
More often with
under a friction’s title names an agent that hits it clearly more often than the others. Choose
Agent behavior
under
Where to fix
, open a friction, and read
Where it happens most
and
Rate by agent
.
Interpret the results
Pick the agent with a high pass rate at a cost you accept, after you check the pair on
Comparisons
.
No clear difference
means that these runs can’t tell the two agents apart, not that they’re the same. Choose on cost or time, or add tasks and trials and launch again.
What you see
What to do
A higher pass rate at a much higher cost
Pay more only for a
Clear difference
.
Run a
Likely difference
again
first.
Similar pass rates, different cost or duration
Pick the one that uses fewer tokens or less time.
One agent fails where others pass
Read its
Agent behavior
frictions and their
Where it happens most
section.
Every agent fails the same task
Suspect the task, the machine, or your product. With
All
selected in Frictions, check
Machine
and
Task quality
first.
From the CLI and the plugin
Your AI assistant can run this guide with the
Obss Plugin
or the
CLI
. Only connecting model providers stays in the dashboard.
Step
CLI
Plugin (MCP)
List agents and their models
obss catalog list --kind agent
inspect_catalog
Launch the plan
obss experiments launch
launch_experiment
Read the Brief
obss experiments brief
get_experiment_brief
Summarize results by agent
obss results analyze summarize_scope
analyze_results
Check which differences hold up
obss evals compare --kind agent
get_evals_comparisons
List one agent’s runs
obss runs list --agent-id
list_runs
List agent behavior frictions
obss frictions list --fix-in agent_behavior
list_frictions
Read where a friction happens most
obss frictions get
get_friction
For example:
obss
frictions
list
--experiment-id
exp_8f2c41d0
--fix-in
agent_behavior
obss
frictions
get
frc_7d20e6b4
--experiment-id
exp_8f2c41d0
Other ways to run it
Start from a template.
The
Latest model shootout
template
on
Home → Templates
runs varied tasks on every agent available to you, with
Raw Agent
and 2 trials.
Compare effort levels.
Run one model at several effort levels to see whether more reasoning pays off.
Compare with your tools.
Add a treatment with your tool to see how much it helps each agent. See
Measure a tool’s impact
.
Add your own agent.
Compare your own ACP agent, connected as a
custom agent
(private beta), with the built-in agents.
Repeat after a new model ships.
Select
Launch again
, then change the models with
Edit first
. See
Track results over time
.
Next steps
Measure a tool's impact
Find out how much your tools help each agent.
Track results over time
Run the comparison again when models change.
Agents
See the agents you can compare and their providers.
Frictions
See why one agent hits a problem more often.
Measure a tool's impact
Track results over time
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform