# Track results over time - Obss

Use cases
Track results over time
Run a fixed task set again after each change, or on a schedule, to catch regressions and drift from product, model, or dependency changes.
Results can shift when your product, a model, an agent, or a dependency changes. Pass rate shows whether results moved, and new frictions show where the agents’ behavior changed.
Prerequisites
A finished experiment with the tasks, treatments, and agents to track, such as one built with
Measure a tool’s impact
or
Compare agents and models
.
Runs for each launch, scheduled or not (
Runs and billing
).
Steps
1
Choose a fixed task set
Pick the tasks that you’ll run every time, and change them as little as possible, because a change to a task changes what you measure.
Tag
them, for example
regression
, so that you can find them later.
2
Run a first experiment
Launch
the tasks with the treatments and agents that you track. You compare later experiments with this one.
3
Launch again after each change
After you change your product, a tool, or a dependency, open the finished experiment and select
Launch again
. It launches a new experiment with the same tasks, treatments, agents, and settings, and each task uses its latest version. If tasks changed, the dialog says how many. To change the model or anything else first, select
Edit first
.
4
Schedule it
To catch changes that you don’t make, such as a new model or a provider update, select
Schedule
on the experiment and choose
Repeat
, the time, and the
Time zone
. The dialog shows the
First launch
and the runs that it
Uses
for each launch and for the next 30 days. Then select
Create schedule
.
5
Compare the experiments
Open the schedule from
Automate → Schedules
. Its
Experiments
tab lists every experiment that it launched,
Runs
lists their runs, and
Frictions
ranks their frictions. Open two experiments and compare their
Matrix
pass rates. In
Analyze → All experiments
, each scheduled experiment shows
From schedule
and the schedule’s name.
6
Look for new frictions
In the schedule’s
Frictions
tab, or in
Analyze → All frictions
, sort the list by
First seen
. A
friction
that first appears after a change points to what the change broke.
Get notified
A scheduled experiment that finishes without problems sends no notification. One with errors or failed Evals notifies you as usual, and Obss also tells you when a schedule needs attention:
Notification
What to do
Schedule paused
: Obss paused the schedule and says why
Fix the problem, then select
Resume
.
Schedule skipping launches
: the last experiment was still running at launch time three times in a row
Choose a less frequent repeat.
Schedule short of runs
: your balance doesn’t cover one of the next launches
Add runs before that launch. The schedule stays on, but Obss pauses it if runs are still short when the launch is due.
These notifications arrive in the app and by email. Turn them on or off with
Experiment updates
under
Profile → Notifications
.
Interpret the results
A stable result keeps about the same pass rate on each task and shows no new frictions.
What you see
What to do
Pass rate drops on several tasks after a change, by more than a trial or two
It’s a regression. Read the frictions that first appeared after the change, and open their runs.
Pass rate moves with no change on your side
It’s drift from a model, an agent release, or a dependency. Compare
Rate by agent
on the new frictions to see whether one agent changed.
Treatment use
drops
Agents call your tool less often. Look for a change in the agent or in your tool’s description.
New
Machine
or
Task quality
frictions
The machine or a task dependency changed. Fix it before you trust the trend.
From the CLI and the plugin
Your AI assistant can do every step with the
Obss Plugin
or the
CLI
.
Step
CLI
Plugin (MCP)
Launch again
obss experiments rerun
rerun_experiment
Preview a schedule
obss schedules preview
preview_schedule
Create a schedule
obss schedules create
manage_schedule
List the experiments it launched
obss experiments list --schedule-id
list_experiments
Rank new frictions
obss frictions list --schedule-id --sort new
list_frictions
Read schedule notifications
obss notifications list
list_notifications
For example, preview and create a schedule for 9:00 AM on weekdays:
obss
schedules
preview
exp_8f2c41d0
--cron
"0 9 * * 1-5"
--time-zone
America/New_York
obss
schedules
create
exp_8f2c41d0
--cron
"0 9 * * 1-5"
--time-zone
America/New_York
To rerun with the first experiment’s task versions instead of the latest, pass
--versions experiment
to
obss experiments rerun
.
Other ways to run it
Launch from a release pipeline.
Run
obss experiments rerun
in your pipeline with a personal API key, then read pass and fail counts with
obss results analyze
. See
Run an experiment in CI
.
Track several task sets.
Use one experiment and one schedule for each set. A project can have up to 50 active schedules.
Compare agents as they change.
Schedule a cross-agent experiment, such as one from the
Latest model shootout
template.
Next steps
Schedules
Set repeats, time zones, and runs for each launch.
Find and fix frictions
Fix a regression, then confirm the fix.
Notifications
Choose what Obss tells you, in the app and by email.
Tags
Label the tasks and experiments that you track.
Compare agents and models
Find and fix frictions
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform