# Experiments - Obss

Run experiments
Experiments
How an experiment turns tasks, treatments, and agents into runs, why the comparison is fair, and how baselines, lift, trials, and statuses work.
Run an experiment to find out which tools and agents do your work best. It runs every combination of the tasks, treatments, and agents you choose, from the same starting point.
What an experiment combines
Each combination is one
run
: one agent, with one treatment, doing one task once.
You choose
What it carries
Tasks
The work: instructions, a rubric, files, and a machine
Treatments (Toolsets)
The tools the agent gets (skills, MCP servers, CLIs, and SDKs), or none
Agents
AI agents such as Claude Code or Codex, built-in or
custom
, each with a model and effort level
Trials
How many times to repeat each combination
Files and a machine attach to a task, tools attach to a treatment, and you choose agents at launch. Learn more in
Where files, machines, and tools attach
.
For example, 3 tasks × 2 treatments × 2 agents × 2 trials makes 24 runs, which uses 24 from your organization’s
run balance
.
Why the comparison is fair
An experiment changes only what you compare. Two treatments on the same task differ only in their tools, and two agents with the same treatment differ only in agent and model.
Same starting point:
every run of a task starts from that task’s files on that task’s machine.
Fresh machine:
each run gets a new machine, so nothing carries over between runs.
Snapshot at launch:
Obss saves the task version, its files and machine, and each treatment’s tools at launch, so later edits don’t change launched runs.
Baseline and lift
The
baseline
is the treatment you compare the others with. It can be any treatment in the experiment. Most experiments use
Raw Agent
, which every project has: it runs the agent as its vendor ships it, with no added tools.
Lift
is the difference in pass rate between a treatment and the baseline, on the same tasks and agents. For example, if agents pass 9 of 20 runs with Raw Agent (45%) and 15 of 20 with your MCP server’s treatment (75%), the lift is 30 percentage points. A small lift doesn’t show that the treatment helped:
Comparisons
tells you whether it holds up.
When you select two or more treatments, you
choose the
Baseline
at launch. Read lift in
Pass rate and lift
.
Trials
Agents don’t give the same result every time, so a
trial
repeats the same inputs. Each trial is a separate run.
Start with two trials. More trials make each pass rate steadier, but they don’t replace tasks: to show that two pass rates really differ,
Comparisons
needs at least 6 tasks where one side did better. Tied tasks don’t count, so write more than 6.
Experiment statuses
The status shows on
All experiments
and on the experiment page, and the CLI and the plugin return its
Value
.
Status
Value
Meaning
Running
running
Runs are still in progress, so the numbers can change as more runs finish.
Cancelling
running
You asked to cancel, and Obss is stopping unfinished work. The cancel call returns
cancellation_requested
.
Completed
completed
All runs finished. The results are final.
Needs review
completed_with_errors
Some runs errored, or Evals or Insights failed for some runs. Review the results before you treat them as final.
Error
errored
An unexpected error stopped the experiment. Try again, and if the error continues, email
support@obss.app
.
Blocked
errored
One or more runs were blocked, so they weren’t scored. Open a run to see the cause and its fix.
Cancelled
cancelled
Someone cancelled it. Completed runs stay available.
Right after you launch, the experiment page can show
Starting your experiment
for a few seconds while Obss prepares the runs. After a launch from the CLI or the plugin,
get_launch_status
returns
admitting
until the experiment exists, or
admission_failed
if the launch didn’t start.
What you can do with an experiment
Action
When
What it does
Launch again
The experiment finished
Launches a new experiment with the same tasks, treatments, agents, and settings.
Edit first
lets you change them.
Schedule
The experiment finished
Launches it again on a repeat, such as every weekday at 9:00 AM.
Cancel experiment
The experiment is running
Stops unfinished runs. Completed runs stay available.
Run evals
or
Rerun evals
Completed or Needs review
Scores the recorded work against the rubric, for example after a rubric change.
Run insights
or
Rerun Insights
Completed or Needs review
Finds frictions in the recorded work.
Share
Any time (a public link needs Completed or Needs review)
Shares the results with a teammate, or publishes a public link.
Delete
The experiment isn’t running
Deletes the experiment, its runs, and any
schedule
that launches it again.
You can start an experiment from
New experiment
, a
template
,
Launch again
, the CLI, or your AI assistant. Every route makes the same kind of experiment, and the results appear in the same place.
Next steps
Launch an experiment
Choose tasks, treatments, agents, and trials, then launch.
Templates
Start from a ready-made experiment.
Read results
Compare Evals results and cost in Matrix.
Schedules
Launch an experiment again on a repeat.
Treatments (Toolsets)
Launch an experiment
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform