# Measure a tool's impact - Obss

Use cases
Measure a tool's impact
Find out whether your skill, MCP server, CLI, or SDK helps agents, whether the gain holds up, and what still blocks them.
Run the same tasks with and without your tool, and compare the pass rates. The difference is the tool’s
lift
.
Prerequisites
A project with at least one
agent
turned on under
Manage → Agents
.
Enough runs in your balance. Each combination of task, treatment, agent, and trial uses one run (
Runs and billing
).
The skill, MCP server, CLI, or SDK that you want to measure.
Steps
1
Write tasks that use your product
In
Manage → Tasks
, write
tasks
like the ones your users give agents, each with a rubric and the
files and machine
it needs. Don’t name your tool in the task, so that you measure whether the agent finds it. Write more than 6 tasks:
Comparisons
needs at least 6 tasks where one side did better, and tied tasks don’t count.
2
Add your tool
In
Manage → Tools
, add the
tool
. Enter any secret it needs, such as an API key, in the dashboard.
3
Create a treatment with your tool
In
Manage → Treatments (Toolsets)
, create a
treatment
that contains your tool. Every project already has
Raw Agent
, the usual baseline, which runs the agent as its vendor ships it.
4
Launch the experiment
Select
New experiment
and choose your tasks, both treatments, and one or more agents. Under
Baseline
, choose
Raw Agent
. Set
Trials
to 2 or more, and
launch
the experiment.
5
Compare pass rates on the Matrix
Open the experiment’s
Matrix
with the
Evals
metric. Compare each of your treatment’s columns with the same agent’s
Raw Agent
column, and hover over a cell for the trial count. For example, 1 of 4 trials passed (25%) on
Raw Agent
and 3 of 4 (75%) on your treatment is a lift of 50 percentage points on that task (
Pass rate and lift
).
Switch the metric to
Treatment use
to see how often the agent called your tool. It’s a lower bound, because calls inside scripts don’t count, and it’s blank for an SDK.
6
Check that the lift holds up
Open the
Comparisons
tab. The sentence at the top says whether your treatment did better than
Raw Agent
, and the block under it gives each side’s pass rate and the result. A
Clear difference
is safe to act on. For a
Likely difference
, select
Run again to be sure · N runs
to launch a small follow-up experiment first. When Insights ran, the
Brief
in the block lists the frictions seen with the side that did worse. Each agent’s own result is under
All N comparisons
. Learn more in
Comparisons
.
7
Find what still blocks agents
On the
Frictions
tab, choose
Product
under
Where to fix
to see changes to make in your tool, API, errors, or docs. Fix the top ones as described in
Find and fix frictions
, then select
Launch again
.
Interpret the results
Your tool helps when
Comparisons
shows a
Clear difference
in your treatment’s favor and
Treatment use
is high.
What you see
What to do
Lift near zero, low
Treatment use
The agent didn’t find the tool. Improve a skill’s description, an MCP server’s tool names, or the CLI’s help.
Lift near zero, high
Treatment use
The agent used the tool, but it didn’t help. Read the product frictions and a few
runs
.
Negative lift with a
Clear difference
The tool made agents worse. Frictions often show errors, wrong results, or misleading docs.
No clear difference
These runs can’t tell the two apart, which doesn’t mean the tool has no effect. Add tasks and trials, then launch again.
Machine
or
Task quality
frictions
The machine or task is at fault, not your tool. Fix these first, because they make the results unreliable.
From the CLI and the plugin
Your AI assistant can run this guide with the
Obss Plugin
or the
CLI
. Only tool secrets stay in the dashboard, and the commands return the link where you enter them.
Step
CLI
Plugin (MCP)
Find tasks, tools, and treatments
obss catalog list
inspect_catalog
Create a tool or treatment
obss resources apply
author_resource
Check and launch the plan
obss experiments preflight
,
obss experiments launch
preflight_experiment
,
launch_experiment
Read the Brief
obss experiments brief
get_experiment_brief
Compare the treatments
obss results analyze compare_treatments
analyze_results
Check that the lift holds up
obss evals compare --kind treatment
get_evals_comparisons
List the experiments that used the tool
obss experiments list --tool-id
list_experiments
List the frictions of the tool
obss frictions list --tool-id
list_frictions
For example, ask your AI assistant:
Launch my checkout tasks with the Raw Agent and Orders MCP treatments on Claude Code, 3 trials each. When it finishes, read the Brief, tell me which differences from Raw Agent hold up, and list the product frictions.
Other ways to run it
Start from a template.
On
Home → Templates
,
Skill lift
runs React tasks with no skill, then with two versions of a skill.
Stripe interface evals
compares Raw Agent with the Stripe MCP server, CLI, and SDK, and with the CLI plus a best-practices skill, all in Stripe test mode. Learn more in
Templates
.
Compare versions of a tool.
Put each version in its own treatment, as the
Skill lift
template does.
Compare interfaces.
Put each interface in its own treatment, as the
Stripe interface evals
template does, to see which gives the most lift for each agent.
Combine tools.
Put a CLI and a skill in one treatment to measure their effect together.
Check every agent.
Add more agents to see whether the tool helps all of them. See
Compare agents and models
.
Next steps
Find and fix frictions
Fix what still blocks agents, then confirm the fix.
Track results over time
Run the same tasks again after each change.
Treatments (Toolsets)
See how treatments group tools.
Read results
Learn how to read pass rate, lift, and the Matrix.
Share results
Compare agents and models
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform