Prerequisites
- The agents to compare, turned on under Manage → Agents, each with a model provider that it supports (Agents lists them). Connect providers under Organization settings → Model providers, or under Profile → Integrations for agents that support a personal login.
- Tasks that stand for your real work.
- Runs in your balance. Each combination of task, treatment, agent, and trial uses one run, and your providers bill the model inference (Runs and billing).
Steps
1
Choose the tasks
Pick tasks of different sizes that cover the work you care about. Pick more than 6: Comparisons needs at least 6 tasks where one agent did better, and tied tasks don’t count.
2
Use one treatment for every agent
Choose Raw Agent to compare the agents as their vendors ship them, or your own treatment to compare them with your product’s tools.
3
Choose agents, models, and effort
Select New experiment. Under Agents, select each agent, its Model, and its effort level. Default effort sends no level, so the agent uses its own default for that model. Set Trials to 2 or more, then launch. You get a notification when it finishes.
4
Compare pass rate with cost
Open the experiment’s Matrix with the Evals metric to see pass rates, then switch to Cost, Tokens, Duration, Steps, or Tool calls to see what each result costs.
5
Check which differences hold up
Open the Comparisons tab. With one treatment, the sentence at the top answers for every pair of agents, and each pair that differs gets a block with each agent’s pass rate and the result. With more than one treatment, every agent pair is under All N comparisons, in the Agents with groups, and a clear difference between agents also gets a block at the top. For a Likely difference, select Run again to be sure · N runs before you choose. When Insights ran, the Brief in a block lists the frictions seen with the agent that did worse.
6
Find out why agents differ
On the Frictions tab, More often with under a friction’s title names an agent that hits it clearly more often than the others. Choose Agent behavior under Where to fix, open a friction, and read Where it happens most and Rate by agent.
Interpret the results
Pick the agent with a high pass rate at a cost you accept, after you check the pair on Comparisons. No clear difference means that these runs can’t tell the two agents apart, not that they’re the same. Choose on cost or time, or add tasks and trials and launch again.From the CLI and the plugin
Your AI assistant can run this guide with the Obss Plugin or the CLI. Only connecting model providers stays in the dashboard.
For example:
Other ways to run it
- Start from a template. The Latest model shootout template on Home → Templates runs varied tasks on every agent available to you, with Raw Agent and 2 trials.
- Compare effort levels. Run one model at several effort levels to see whether more reasoning pays off.
- Compare with your tools. Add a treatment with your tool to see how much it helps each agent. See Measure a tool’s impact.
- Add your own agent. Compare your own ACP agent, connected as a custom agent (private beta), with the built-in agents.
- Repeat after a new model ships. Select Launch again, then change the models with Edit first. See Track results over time.
