# obss experiments - Obss

CLI
obss experiments
Write a launch plan, check it, launch an experiment, watch its progress, read its Brief, launch it again, cancel it, and publish its results from the CLI.
An experiment runs every combination of its tasks, treatments, and agents once per trial, and each of those is one run. These commands launch experiments from a JSON plan, follow them, and share their results. To decide what goes in one, see
Launch an experiment
.
Launch plan
A launch plan is one JSON object that names every task version, treatment, and agent, so the same plan always launches the same work.
preflight
and
launch
both read it, and
obss schema experiments launch
lists every field and limit.
To build a plan, run
obss experiments clone-plan
on an experiment you already ran, and edit it. For your first experiment, copy the example below and replace its IDs with the IDs from
obss catalog list --projection full
. For a baseline, use the
trt_
ID of the project’s
Raw Agent
treatment.
Example
This plan runs 2 tasks × 2 treatments × 2 agents × 2 trials, which is 16 runs:
{
"label"
:
"Checkout: payments MCP vs baseline"
,
"selectedTaskDefinitionIds"
: [
"tskver_3a91c7e2"
,
"tskver_b5047d1e"
],
"treatments"
: [
{
"clientId"
:
"baseline"
,
"name"
:
"Baseline"
,
"treatmentDefinitionId"
:
"trt_51be0c9a"
},
{
"clientId"
:
"payments-mcp"
,
"name"
:
"Payments MCP"
,
"treatmentDefinitionId"
:
"trt_0c6d2a94"
,
"mcpServerVersionIds"
: [
"mcpver_a7d3c019"
]
}
],
"referenceTreatmentClientId"
:
"baseline"
,
"agentSelections"
: [
{
"projectAgentModelBindingId"
:
"agm_7b2d4e10"
},
{
"projectAgentModelBindingId"
:
"agm_19f6c8a3"
,
"effortLevel"
:
"high"
}
],
"projectConfigurationRevision"
:
42
,
"organizationConfigurationRevision"
:
17
,
"trialCount"
:
2
,
"maxParallelAgentSessions"
:
4
,
"evals"
:
true
,
"insights"
:
true
,
"launchAttemptId"
:
"checkout-payments-mcp-2026-11-12"
}
Fields
Field
Required
Description
selectedTaskDefinitionIds
Yes
1 to 100 task version IDs: each task’s
tskver_
versionId
from
obss catalog list --kind task
, not its
tsk_
ID.
treatments
Yes
1 to 100 distinct treatments, each with a
clientId
(your own label for this plan), a
name
, and the saved treatment’s
trt_
treatmentDefinitionId
.
treatments[].skillVersionIds
,
mcpServerVersionIds
,
cliVersionIds
,
sdkVersionIds
No
The tool versions the agent gets in this treatment, up to 100 of each kind. Leave them out or empty for a baseline.
referenceTreatmentClientId
Yes
The
clientId
of the Baseline, one of the plan’s treatments and usually
Raw Agent
.
obss results analyze compare_treatments
reads every other treatment against it.
agentSelections
Yes
1 to 100 agents, each with a
projectAgentModelBindingId
: an agent’s
agm_
ID from
obss catalog list --kind agent --projection full
.
agentSelections[].effortLevel
No
The effort level for that agent and model. Leave it out for
Default
, which sends no level. See
Agents
.
projectConfigurationRevision
,
organizationConfigurationRevision
Yes
The configuration versions in an agent’s
facts
from
obss catalog list --projection full
. If agents, models, or providers changed since, the launch fails with
configuration_changed
. Preflight returns the current values in its
plan
.
trialCount
Yes
Trials of each combination, 1 to 20. Each trial is one run from your balance.
maxParallelAgentSessions
No
Max Concurrency
: agent sessions active at once, 1 to 100,000;
1
runs them one at a time. Without it, the organization limit applies. Obss can run fewer to protect capacity. It doesn’t affect Evals or Insights.
evals
No
Run
Evals
on each run. Defaults to
true
.
evalsMode
No
standard
(the default) or
agent
.
agent
needs
evals: true
and uses
Agent mode
(private beta).
insights
No
Run
Insights
on each run. Defaults to
true
.
insightsMode
No
standard
(the default) or
agent
.
agent
needs
insights: true
and uses
Agent mode
(private beta).
projectAiSelections
No
Models for this experiment only, up to one per role:
judge
(the Evals model, needs
evals: true
), and
sweep
(the Insights model) and
consolidation
(the Experiment insights model), which need
insights: true
. Each has a
providerConnectionId
(
aic_
), a
providerModelId
(
aim_
), and an optional
effortLevel
. A role you leave out uses the organization’s model.
projectAiInstructionOverrides
No
Custom instructions for this experiment only: one
{ "role", "customInstructions" }
per role.
label
No
The experiment’s name, up to 120 characters.
description
No
The experiment’s description.
launchAttemptId
Yes
An ID you choose for this launch, up to 100 characters. Send the same plan with the same ID to retry without a second launch.
A plan can hold at most 100,000 runs. Keep any extra fields that
clone-plan
adds, such as a treatment’s
contentHash
.
obss experiments preflight
Check a launch plan without launching it or spending runs.
obss
experiments
preflight
--plan-file
<
pat
h
>
[flags]
Plugin (MCP) tool:
preflight_experiment
Flags
Flag
Type
Default
Description
--plan
string
The plan as inline JSON.
--plan-file
file path
A file that holds the plan JSON.
--require-ready
boolean
false
Exit with code
6
when the plan is not ready. The output does not change.
Pass exactly one of
--plan
or
--plan-file
. Also takes the
global flags
.
Examples
obss
experiments
preflight
--plan-file
./experiment.json
--output
json
obss
experiments
preflight
--plan-file
./experiment.json
--require-ready
Result fields
Field
Description
ready
true
when nothing blocks the launch.
runCount
The runs the launch uses.
blockers
What stops the launch, each with a
code
, a
message
, and a
setupUrl
when the fix is on the web. Examples: a disconnected provider, or
variable_name_clash
for a
variable that two places set to different values
.
issues
Each machine or tool that still needs a required variable, header, or OAuth connection, with
missingFields
and the
setupUrl
where a person enters them. Variables not marked required aren’t checked.
notices
Warnings that do not block, such as a custom agent that needs attention or was not tested. Each has a
url
.
quota
requiredRuns
,
availableRuns
, and
covered
: whether your balance covers the launch.
agentModeEstimates
Only for a feature that the plan runs in Agent mode while your organization can use it: one entry per feature, with
feature
(
evals
or
insights
) and
estimate
: the
runs
its agent would review, the
model
it uses, and for Insights
experimentModel
when the Experiment insights model is a different model. The terminal output doesn’t print it.
plan
Your plan, with the current configuration versions filled in.
Notes
Preflight exits
0
whether or not the plan is ready, so a script either checks
ready
or passes
--require-ready
. A plan with
evalsMode
or
insightsMode
set to
agent
gets the blocker
agent_mode_unavailable
when your organization can’t use that
Agent mode
, or
agent_mode_model_unsupported
, with a
setupUrl
to
Obss AI features
, when a model can’t run it. A plan that doesn’t match the schema fails with a usage error (exit code
2
) that names the first wrong fields, and
details
lists every issue.
obss experiments launch
Launch an experiment from a plan.
obss
experiments
launch
--plan-file
<
pat
h
>
[flags]
Plugin (MCP) tool:
launch_experiment
This command spends runs from your organization’s balance and doesn’t ask you to confirm, so run
preflight
first. In Ask Obss, you approve each call.
Flags
Flag
Type
Default
Description
--plan
string
The plan as inline JSON.
--plan-file
file path
A file that holds the plan JSON.
Pass exactly one of
--plan
or
--plan-file
. Also takes the
global flags
.
Examples
Launch, then watch until the experiment ends:
obss
experiments
launch
--plan-file
./experiment.json
--output
json
obss
experiments
watch
adm_6c1e93b7
Result fields
Field
Description
launchId
The
adm_
launch ID.
experimentId
The
exp_
experiment ID, or
null
until Obss admits the launch.
status
accepted
.
runCount
The runs the launch uses.
dashboardUrl
The experiment’s page in the dashboard.
Notes
obss experiments status
and
obss experiments watch
take either ID. To retry after a lost response, send the same plan with the same
launchAttemptId
, and you get the first launch back. A refused launch exits with code
4
, and
error.code
is one of these:
Code
Meaning
What to do
quota_exceeded
,
no_entitlement
,
grant_expired
Your run balance does not cover the launch. Preflight shows the same in
quota
.
A person fixes it on
Organization settings → Usage & billing
. A new attempt ID does not help.
configuration_changed
,
version_selection_changed
An agent, model, provider, or tool version changed after you read the plan.
Run preflight again and launch the
plan
it returns, with a new
launchAttemptId
.
launch_attempt_spent
The
launchAttemptId
belongs to an earlier launch that failed or expired.
Launch again with a new
launchAttemptId
.
launch_attempt_conflict
The same
launchAttemptId
came with a different plan.
Use a new
launchAttemptId
for the new plan.
In a pipeline, build a new
launchAttemptId
for each launch, such as from the pipeline run number, and reuse it only to retry that launch. A saved plan file keeps its old
launchAttemptId
and revisions, so launch the
plan
that preflight returns, not the file.
obss experiments status
Read an experiment’s progress once.
obss
experiments
status
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
get_experiment_status
. For an
adm_
launch ID:
get_launch_status
.
Arguments
Argument
Description
<experiment-id>
Required. The
exp_
experiment ID, or the
adm_
launch ID from a launch.
Flags
Flag
Type
Default
Description
--wait-ms
integer
Wait in the server up to this many milliseconds, at most 25,000. The read returns when the status changes, a run finishes, or the experiment ends.
Also takes the
global flags
.
Examples
obss
experiments
status
exp_8f2c41d0
--wait-ms
25000
--output
json
Result fields
Field
Description
experiment.status
admitting
while Obss admits a launch, then
running
. It ends as
completed
,
completed_with_errors
,
cancelled
,
errored
, or
admission_failed
.
experiment.error
On a failed experiment, a
message
and a
reference
to give support.
progress
The run counts in
Progress counts
.
terminal
true
once the experiment has ended.
pollAfterMs
How long to wait before the next read.
Notes
When
terminal
is true, read the final runs with
obss runs list
. If you run
obss runs list
while a
--wait-ms
status read is still waiting, it can answer first with an older state.
obss experiments watch
Wait until an experiment ends, then print its final status.
obss
experiments
watch
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
get_experiment_status
,
get_launch_status
. MCP hosts poll the status with
wait_ms
.
Arguments
Argument
Description
<experiment-id>
Required. The
exp_
experiment ID, or the
adm_
launch ID.
Flags
Flag
Type
Default
Description
--timeout-seconds
integer
3600
Stop waiting after this many seconds, at most 86,400.
--fail-on
string
End statuses that make the command exit with code
6
, separated by commas:
completed_with_errors
,
cancelled
,
errored
,
admission_failed
.
Also takes the
global flags
.
Examples
obss
experiments
watch
adm_6c1e93b7
--timeout-seconds
7200
obss
experiments
watch
adm_6c1e93b7
--fail-on
completed_with_errors,cancelled,errored,admission_failed
Result fields
When
terminal
is true, the command returns the same fields as
obss experiments status
:
experiment
,
progress
,
terminal
, and
pollAfterMs
.
Notes
Each read waits in the server for a change, and the command prints a progress line on stderr after each one unless you pass
--quiet
. Right after a launch, it also waits the few seconds an experiment can take to appear.
When
Exit code
The experiment ends in a status that
--fail-on
doesn’t name, or you didn’t pass
--fail-on
0
. Only
completed
means every run finished, so read
data.experiment.status
.
The experiment ends in a status that
--fail-on
names
6
, with the same result
--timeout-seconds
passes first
4
, with
error.code
watch_timeout
On a timeout, the message is
Experiment did not reach a terminal state after <n> seconds.
, and
error.details
has the
id
you watched, the
experimentId
when known, and the last
status
read. The experiment keeps running; watch it again. For pass counts, run
obss results analyze
after the watch ends.
obss experiments list
List and filter the experiments in the project.
obss
experiments
list
[flags]
Plugin (MCP) tool:
list_experiments
Flags
Flag
Type
Default
Description
--schedule-id
string
Only the experiments that this
sch_
schedule launched.
--task-id
string
Only the experiments with runs of this
tsk_
task, in every version.
--task-version-id
string
Only the experiments with runs of this
tskver_
task version.
--treatment-id
string
Only the experiments with runs of this
trt_
treatment (toolset).
--tool-id
string
Only the experiments with runs whose treatment gave the agent this skill, MCP server, CLI, or SDK (
skl_
,
mcp_
,
cli_
, or
sdk_
), in any version.
--tool-version-id
string
Only the experiments with runs whose treatment gave the agent this tool version (
sklver_
,
mcpver_
,
cliver_
, or
sdkver_
).
--file-id
string
Only the experiments with runs whose task had this
fld_
file.
--machine-id
string
Only the experiments with runs whose task ran on this
env_
machine.
--agent-id
string
Only the experiments with runs of this agent key, such as
claude-code
, with any model.
--agent-model-id
string
Only the experiments with runs of this
agm_
agent and model.
--friction-id
string
Only the experiments with runs where this
frc_
friction occurred.
--query
string
Search text, such as part of an experiment’s name.
--status
running
,
completed
,
completed_with_errors
,
cancelled
,
errored
Only experiments with this status.
--created-after
string
Only experiments created after this ISO 8601 date-time.
--created-before
string
Only experiments created before this ISO 8601 date-time.
--tag
string
Only experiments with any of these comma-separated tags, by name or
tag_
ID.
--untagged
boolean
false
Only experiments with no tags. With
--tag
, experiments with those tags or with none.
--based-on-experiment-id
string
Only the child experiments of this
exp_
parent experiment: launched again, edited, or scheduled from it.
--limit
integer
50
Results per page, up to 200 (default 50).
--cursor
string
The
nextCursor
from the previous page.
The first eleven flags are the
run scope flags
except
--experiment-id
. Also takes the
global flags
.
Examples
List the experiments tagged for two releases:
obss
experiments
list
--tag
release-11,release-12
List what a schedule launched:
obss
experiments
list
--schedule-id
sch_c0a9f311
--output
json
List the experiments whose runs gave the agent an MCP server:
obss
experiments
list
--tool-id
mcp_2e8b6f13
Result fields
Field
Description
experimentId
,
label
,
status
The experiment’s ID, name, and status.
Counts
Its tasks, runs, agents, treatments, and trials.
infraBlockedCount
Runs that completed or errored while blocked, so Evals didn’t check them.
tags
,
lineage
,
dashboardUrl
Its tags, where it came from (as in
obss experiments get
), and its dashboard page.
hasMore
,
nextCursor
Whether another page exists, and the cursor for it.
Unlike
obss runs list
, the result has no
scope
.
Notes
A run scope flag lists the experiments that have at least one run in that scope: the experiments of the runs that
obss runs list
returns with the same flags.
--schedule-id
also lists the experiments that the schedule launched before they have runs. An ID that the project doesn’t have fails with
scope_not_found
, and an unknown tag fails with
tag_not_found
, with
details.tags
naming it.
obss experiments get
Read an experiment’s progress, where it came from, who launched it, and the runs it used.
obss
experiments
get
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
get_experiment
Arguments
Argument
Description
<experiment-id>
Required. The
exp_
experiment ID.
Takes only the
global flags
.
Examples
obss
experiments
get
exp_8f2c41d0
Result fields
Field
Description
experiment
The experiment, with its
lineage
.
progress
The run counts in
Progress counts
.
facts.launchedBy
Who launched it.
facts.quota
The runs it used, and the balance left when it finished.
experiment.lineage.launchedVia
new
,
from_experiment
,
schedule
, or
schedule_launch_now
.
experiment.lineage.basedOn
Its parent experiment: the experiment it was launched from.
experiment.lineage.schedule
The schedule that launched it, and the time it was scheduled for.
experiment.lineage.children
Counts of its child experiments and of the schedules that launch it.
Notes
A deleted experiment or schedule in
lineage
shows as deleted.
obss experiments brief
Read an experiment’s
Brief
: each clear or likely difference as
Comparisons
shows it, with one written sentence and the frictions seen with the side that did worse.
obss
experiments
brief
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
get_experiment_brief
Arguments
Argument
Description
<experiment-id>
Required. The
exp_
experiment ID.
Flags
Flag
Type
Default
Description
--issue
string
The
cmp_
ID of one difference in this Brief: print it as an issue in Markdown, the text that
Copy as issue
copies. With
--output json
, every finding carries it as
issueMarkdown
.
Also takes the
global flags
.
Examples
Read the Brief:
obss
experiments
brief
exp_8f2c41d0
Stripe CLI did better than Raw Agent on all 7 tasks.
Stripe CLI and Raw Agent  cmp_4a0e2c9d
Stripe CLI   93%   26 of 28 runs
Raw Agent     0%    0 of 28 runs
Better on all 7 tasks.
Clear difference
Both agents did better with Stripe CLI.
Brief: Raw Agent runs had no authenticated Stripe access, and the task asked for access that the setup did not give.
Frictions on the 7 tasks where Raw Agent did worse
Missing authenticated Stripe access · Setup  Raw Agent 24 of 28 failed runs · Stripe CLI 0 of 26 passed runs  frc_3c9a1e07
Baseline task and access contract mismatch · Setup  Raw Agent 20 of 28 failed runs · Stripe CLI 0 of 26 passed runs  frc_6b2d8f40
https://app.obss.app/projects/prj_2b7c9e14/experiments/exp_8f2c41d0/comparisons
Notes
With 7 tasks, a result for one agent alone or between agents can be a likely difference at most.
In 2 of 28 Matrix cells, repeated runs got different results.
Frictions listed under a result were seen with the side that did worse. That does not mean they caused it.
It prints the sentence that opens
Comparisons
, then each difference: its block as
obss evals compare
prints it,
Brief:
and the sentence, and one line per friction with the failed runs of the side that did worse and the passed runs of the other side. A
Likely difference
prints its next step and command on its result line, and a
Clear difference
with a friction to fix in the product prints
After a fix, run the N tasks that differed again · N runs
with its command. After the differences come the state message, if any, what the differences leave out, the link to
Comparisons
, and
Notes
.
With Insights in
Agent mode
, each difference ends with
What the runs did differently
. Each task has one line for the failed run and one for the passed run, with the step lines they quote and the run IDs. Once the insights agent has read every pair, the
Brief:
sentence says what the runs did across the tasks, and the command prints the example task with “Like N other tasks”, then each task that went differently with “Went differently”. Before that, it prints the first three tasks, and “Reading N more tasks…” while the agent is still reading. Last, it says how many tasks are left, with the
--output json
command that returns them all:
Brief: Raw Agent runs found no Stripe CLI or Stripe key, while Stripe CLI runs found the CLI and a test key.
...
What the runs did differently · 1 of 7 tasks
Create a customer with metadata
Like 6 other tasks
Raw Agent · failed run: The run checked for a stripe binary and STRIPE env vars. Neither appeared, and the output showed only the /app listing.
Step 4: "which stripe 2>/dev/null; env | grep -i stripe ..."  run_c0fdda55 trial run_4ae0a340
Stripe CLI · passed run: The run checked for a stripe binary and STRIPE env vars. It found /usr/local/bin/stripe and STRIPE_API_KEY=sk_test_.
Step 4: "which stripe 2>/dev/null; env | grep -i stripe ... output: /usr/local/bin/stripe STRIPE_API_KEY=sk_test_***REDACTED***"  run_94fd9a78 trial run_9567fde4
6 more tasks: obss experiments brief exp_8f2c41d0 --output json
Print one difference as an issue in Markdown, the text that
Copy as issue
copies:
obss
experiments
brief
exp_8f2c41d0
--issue
cmp_4a0e2c9d
Read the differences and their counts as JSON:
obss
experiments
brief
exp_8f2c41d0
--output
json
Result fields
Field
Description
state
How complete the Brief is. See
Notes
.
message
The
title
and
description
for the state.
null
when
state
is
ready
.
briefId
The
brf_
ID. Set only when
state
is
ready
or
failed
.
summary
The sentence that opens
Comparisons
, the same as
summary
of
obss evals compare
and the
Matrix
line:
text
, and
parts
with each treatment’s or agent’s key. “Nothing to compare in this experiment.” or “This experiment is too large to compare.” when there are no comparisons.
null
when the experiment has no main comparison, or when
state
is
no_insights
or
not_available
.
findings
The differences, at most 5, as below. Empty when
state
is
no_insights
or
not_available
.
comparisonsLeftOut
Clear and likely comparisons that aren’t findings, such as one agent’s row of a difference that the Brief shows for all agents. Read them with
obss evals compare
.
restLine
What the differences leave out, as in
obss evals compare
notes
What limits the comparisons, as in
obss evals compare
evalsId
,
insightsId
The latest Evals and Insights attempts when the findings were read
dashboardUrl
The experiment’s
Comparisons
tab, where the Brief shows
waitMs
Only while
state
is
writing
, or while the insights agent is still reading some pairs of runs: read again after this many milliseconds.
hasMore
true
when the read left out some pairs of runs in
explanations
to fit its size limit. The field is absent otherwise.
error
Only when
state
is
failed
:
code
and
message
. See
Notes
.
Each finding has these fields:
Field
Description
comparisonId
,
kind
,
label
,
title
,
scope
The comparison, as in
obss evals compare
.
kind
is
treatment
or
agent
, and
label
is
clear
or
likely
.
name
,
sides
,
taskLine
,
verdict
,
nextStep
,
byAgent
,
headline
Its block, in the words of
obss evals compare
arm
,
reference
The two sides, each with
key
,
label
, and the
passes
and
runs
it compared
tasks
,
pValue
,
adjustedPValue
As in
obss evals compare
worseSide
arm
or
reference
: the side that did better on fewer tasks
worseTaskCount
The tasks where
worseSide
did worse. The friction counts use only these tasks.
notReviewed
For
arm
and
reference
, the compared runs on those tasks that Insights didn’t review. No count includes them.
sentence
The written sentence.
null
unless
state
is
ready
.
confirmation
,
confirmationUnavailable
As in
obss evals compare
. Plan the runs of a likely difference with
obss experiments clone-plan --comparison
.
frictions
Up to 3 frictions seen with
worseSide
, the largest gap first. Each has
frictionId
(
frc_
),
title
,
summary
(what goes wrong, in one line),
fixIn
,
tasks
,
dashboardUrl
,
failedRuns
and
passedRuns
(how many of the worse side’s failed runs, and of the other side’s passed runs, had it:
runsWithFriction
out of
runs
), and, for
arm
and
reference
,
runsWithFriction
out of
runsReviewed
, which is how the friction was picked.
afterFix
For a clear difference with a friction to fix in the product:
text
(“After a fix, run the 7 tasks that differed again · 56 runs”),
taskCount
, and
runs
. Plan the runs with
obss experiments clone-plan --comparison … --after-fix
.
null
otherwise.
issueMarkdown
The difference as an issue in Markdown, the text that
--issue
prints and
Copy as issue
copies
explainedTaskCount
Agent mode only: how many tasks have a pair of runs to read side by side, a failed run of
worseSide
and a passed run of the other side
explanations
Agent mode only: every pair of runs, as below. Once
exampleTaskVersionId
is set, the example comes first, then the tasks in
differentTaskVersionIds
, then the rest. Before that, the largest pass-rate gap comes first.
pendingTaskCount
Agent mode only: how many pairs the insights agent is still reading. The Brief is ready before they are.
exampleTaskVersionId
Agent mode only, once every pair is read and
sentence
says what the runs did across the tasks: the
tskver_
task whose pair shows it. Show it with “Like N other tasks”.
differentTaskVersionIds
With
exampleTaskVersionId
: the tasks that went differently, which
sentence
names, in order. Show each with “Went differently”.
similarTaskCount
With
exampleTaskVersionId
: how many other tasks with lines went like the example
Each pair in
explanations
has these fields:
Field
Description
taskVersionId
,
taskName
,
dashboardUrl
The task version that both runs ran (
tskver_
), its name, and its page in this experiment
state
pending
while the agent reads the pair,
completed
when its lines are written, or
failed
when the agent couldn’t explain the pair. A
failed
pair shows only the two runs.
worse
,
better
A failed run of
worseSide
and a passed run of the other side, as below
worse
and
better
each have these fields:
Field
Description
runId
,
trialId
The run, and the trial of it that was read
sideRuns
,
sideMatched
The side’s compared runs on the task, and how many of them failed (
worse
) or passed (
better
)
sideCountLine
”Raw Agent failed 2 of 3 runs” when the side’s runs disagree, or
null
explained
stepIndex
,
line
(what the run did at that step, in one plain line), and
quote
(up to 5 lines of that step, as the run’s trace shows them).
null
unless the pair is
completed
.
dashboardUrl
The run’s trace, at that step when there is one
Notes
The command changes nothing. Obss chooses the differences and counts every number from the experiment’s Evals verdicts and frictions. A model writes only each
sentence
and, in Agent mode, each pair’s
line
. A friction listed with a difference was seen with the side that did worse. That does not mean it caused the difference.
In Agent mode, a pair says where two runs went different ways, not why the difference happened. What the insights agent used to read the pairs counts in the Insights attempt’s
agentUsage
(
obss insights list
).
state
What the read has
message
What to do
ready
The differences, each with its
sentence
null
In Agent mode, read again after
waitMs
while a finding’s
pendingTaskCount
is above 0.
writing
The differences and counts, while the Brief is being written
”Writing the Brief. The results are ready now.”
Read again after
waitMs
.
failed
The differences and counts, with no
sentence
”The Brief could not be written.”, then the reason for
error.code
See the
error.code
table.
not_written
The differences and counts, with no Brief for these results
”No Brief yet. This experiment gets a Brief the next time its Evals or Insights finish.”
Nothing. The counts are complete.
no_insights
No findings
”No Brief. Insights has not reviewed this experiment’s runs. Run Insights to get one.”
Run Insights.
not_available
No findings
”No Brief yet. The Brief is written when the experiment, its Evals, and its Insights finish.”
Read again when they finish.
error.code
error.message
brief_model_unavailable
”The Experiment insights model could not be used. Check it in Obss AI features.” The person who started the Evals or Insights can’t use the model’s provider.
brief_model_refused
”The model declined to write it.”
brief_write_failed
”It is written again the next time Evals or Insights finish.”
The findings are the differences that
Comparisons
shows as blocks. A
No clear difference
, a result for whether a treatment helps one agent more, and one agent’s row of a difference already shown for all agents are never findings.
A friction is listed with a difference only when at least 3 counted runs of
worseSide
have it, on at least 2 tasks, at a higher rate than the other side. See
Brief
.
In Agent mode, only runs that Evals marked Pass or Fail are paired, one pair per task.
Public experiment links
don’t show the Brief.
obss experiments clone-plan
Get a launch plan from an experiment, ready to edit and launch.
obss
experiments
clone-plan
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
get_experiment_plan
Arguments
Argument
Description
<experiment-id>
Required. The
exp_
experiment to copy.
Flags
Flag
Type
Default
Description
--versions
latest
,
experiment
latest
latest
uses each task’s latest version.
experiment
keeps the versions the experiment ran. With
--comparison
, the default is
experiment
.
--comparison
string
The
cmp_
ID of a likely difference from
obss evals compare
: plan only the runs that check it again, from the latest Evals. See
Notes
.
--after-fix
boolean
false
With
--comparison
of a clear difference that lists a friction to fix in the product (its
afterFix
in
obss experiments brief
): plan the runs that check it again after your fix, on the tasks where its two sides differed, with both sides and the experiment’s runs each.
--attempt-id
string
An ID you choose, so that a retry doesn’t repeat the change. The CLI writes it to the plan’s
launchAttemptId
. Up to 100 characters. Without it, the CLI makes a new one.
Also takes the
global flags
.
Examples
Save a plan, change it, check it, and launch it:
obss
experiments
clone-plan
exp_8f2c41d0
--output
json
\
|
jq
'.data.plan'
>
experiment.json
obss
experiments
preflight
--plan-file
./experiment.json
obss
experiments
launch
--plan-file
./experiment.json
Plan the runs that check one likely difference again, check them, and launch them:
obss
experiments
clone-plan
exp_8f2c41d0
\
--comparison
cmp_4a0e2c9d
--output
json
\
|
jq
'.data.plan'
>
run-again.json
obss
experiments
preflight
--plan-file
./run-again.json
obss
experiments
launch
--plan-file
./run-again.json
Result fields
Field
Description
plan
The launch plan, ready for
preflight
and
launch
.
runCount
The runs the plan uses.
sourceExperiment
The experiment the plan came from.
versionMode
latest
or
experiment
.
versionChanges
Each task whose version differs from the experiment’s.
Notes
The command changes nothing. Text output only counts the version changes, so use
--output json
to get the plan.
--comparison
takes the
cmp_
ID of a
Likely difference
from
obss evals compare
, and the plan runs that likely difference again. Launching the plan uses runs from your run balance.
Tasks:
The plan uses the comparison’s tasks, except tasks that already passed in every run, or failed in every run, at least 4 times on both sides together.
Sides:
A treatment comparison keeps its agent, or every agent for all agents together. An agent comparison keeps its treatment. A comparison of whether a treatment helps one agent more (
interaction
) keeps both treatments and both agents.
Runs each:
Each combination gets enough runs that the new experiment will likely
settle the difference
: from the experiment’s own (at least 3) up to 10.
Versions:
The plan keeps the experiment’s task versions unless you pass
--versions latest
. Treatments use their current tools, as with
Launch again
.
Evals:
Evals is on, with the mode, Evals model, and instructions of the experiment’s latest Evals attempt (the one that found the difference), even when the experiment launched with Evals off.
Name:
“Run again:”, followed by the two sides and what they share, for example “Run again: Stripe CLI vs Raw Agent, Codex · gpt-5.5 · high”. For all agents together, the name ends with the two sides.
Refusal
When
comparison_not_found
The ID names no comparison in the experiment’s latest Evals.
comparison_not_confirmable
details.reason
says why:
clear
or
unclear
(not a likely difference),
earlier_evals
(a likely difference is run again from the latest Evals),
in_progress
(the experiment or its Evals is still running), or
cannot_settle
(running these tasks again can’t settle it: too few of them can still show a difference, or it would take more than 10 runs each).
rerun_selection_unavailable
A task, treatment, or model of the comparison is no longer available.
obss experiments rerun
Launch a finished experiment again in one call.
obss
experiments
rerun
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
rerun_experiment
This command spends runs from your organization’s balance. In Ask Obss, you approve each call.
Evals and Insights run in the modes the experiment launched with, including
Agent mode
. If your organization can no longer use that Agent mode, the launch is refused with
agent_mode_unavailable
. To launch in Standard mode, plan the launch with
clone-plan
and set
evalsMode
or
insightsMode
to
standard
.
Arguments
Argument
Description
<experiment-id>
Required. The finished
exp_
experiment to launch again.
Flags
Flag
Type
Default
Description
--versions
latest
,
experiment
latest
Task versions:
latest
, or the experiment’s own.
--attempt-id
string
An ID you choose, so that a retry doesn’t repeat the change. Up to 100 characters. Without it, the CLI makes one and prints it on stderr.
Also takes the
global flags
.
Examples
obss
experiments
rerun
exp_8f2c41d0
--attempt-id
rerun-checkout-2026-11-12
--output
json
Notes
The new experiment has the same name, tasks, treatments, agents, trial count, Evals, and Insights, with each treatment’s current saved tools. If a task or treatment it used was deleted, the command refuses rather than launch less. To change the plan first, use
clone-plan
, edit it, and
launch
it.
Result fields
The same fields as
obss experiments launch
:
launchId
,
experimentId
,
status
,
runCount
, and
dashboardUrl
. The text output also prints the
obss experiments watch
command for the launch.
obss experiments cancel
Stop the unfinished work of an experiment, or of one run.
obss
experiments
cancel
<
experiment-i
d
>
[flags]
Plugin (MCP) tool:
cancel_experiment
Cancelled runs stop and don’t finish. Completed work stays available.
Arguments
Argument
Description
<experiment-id>
Required. The
exp_
experiment ID, or the
adm_
launch ID.
Flags
Flag
Type
Default
Description
--run-id
string
A
run_
ID. Cancels every unfinished trial of the same task, treatment, and agent.
--yes
boolean
false
Confirm without a prompt. Required with
--output json
or without a terminal.
Also takes the
global flags
.
Examples
obss
experiments
cancel
exp_8f2c41d0
--yes
--output
json
Result fields
Field
Description
status
cancellation_requested
or
run_cancellation_requested
when the command stopped work.
already_terminal
or
run_already_terminal
when the experiment or run had already finished or was already stopping, so nothing changed.
unitCount
With
--run-id
, the trials it cancelled.
obss experiments publish
Publish a completed experiment at a public link that anyone can open without an account.
obss
experiments
publish
<
experiment-i
d
>
--title
<
titl
e
>
[flags]
Plugin (MCP) tool:
publish_experiment
This command makes results public. In Ask Obss, you approve each call.
Arguments
Argument
Description
<experiment-id>
Required. The completed
exp_
experiment to publish.
Flags
Flag
Type
Default
Description
--title
string
Required. The public title, up to 120 characters.
--description
string
The public description, up to 2,000 characters.
--show-organization
boolean
false
Show your organization’s name on the public page.
--attempt-id
string
An ID you choose, so that a retry doesn’t repeat the change. 16 to 200 characters. Without it, the CLI makes a new one for each call.
Also takes the
global flags
.
Examples
obss
experiments
publish
exp_8f2c41d0
--title
"Payments MCP vs baseline, November"
--show-organization
--output
json
Result fields
Field
Description
publication.publicationId
The
pub_
public link ID.
publication.publicUrl
The public link.
publication.status
preparing
(
Publishing
) while Obss builds the page, then
active
(
Live
). Otherwise
failed
(
Publish failed
) or
revoked
(
Unpublished
).
publication.lockVersion
The value to pass as
--lock-version
on the next change.
Notes
The page includes runs, traces, outputs, Evals, Insights, and text files up to 256 KiB. Larger files, binary files, and private machine settings aren’t shared. See
Share results
.
Later results go live only when you
publish an update
.
obss experiments publications list
List your organization’s public links.
obss
experiments
publications
list
[flags]
Plugin (MCP) tool:
list_experiment_publications
Flags
Flag
Type
Default
Description
--project-id
string
Only the links of this
prj_
project. Without it, every project in the organization.
--experiment-id
string
Only the link of this
exp_
experiment.
--status
preparing
,
active
,
failed
,
revoked
Only links with this status.
--limit
integer
50
Results per page, up to 200 (default 50).
--cursor
string
The
nextCursor
from the previous page.
Also takes the
global flags
.
Examples
obss
experiments
publications
list
--status
active
obss experiments publications get
Read one public link, its status, and the version it shows.
obss
experiments
publications
get
<
publication-i
d
>
[flags]
Plugin (MCP) tool:
get_experiment_publication
Arguments
Argument
Description
<publication-id>
Required. The
pub_
public link ID.
Takes only the
global flags
.
Examples
Read the
lockVersion
before you update or unpublish:
obss
experiments
publications
get
pub_2b8e5f17
--output
json
Result fields
Field
Description
publication.lockVersion
The value to pass as
--lock-version
to update or unpublish.
publication.updateAvailable
true
when the experiment has results that the page doesn’t show yet.
publication.snapshot.version
The version the page shows.
pendingBuildId
Set while an update is being prepared.
obss experiments publications update
Publish the latest results to an existing public link.
obss
experiments
publications
update
<
publication-i
d
>
--lock-version
<
n
>
[flags]
Plugin (MCP) tool:
update_experiment_publication
This command makes results public. In Ask Obss, you approve each call.
The update replaces the results at the public link. The current page stays live until the new version is ready.
Arguments
Argument
Description
<publication-id>
Required. The
pub_
public link ID.
Flags
Flag
Type
Default
Description
--lock-version
integer
Required. The
lockVersion
from
obss experiments publications get
. When the link changed since, the update fails with
publication_changed
.
--attempt-id
string
An ID you choose, so that a retry doesn’t repeat the change. 16 to 200 characters. Without it, the CLI makes a new one for each call.
--retry-pending
boolean
false
Take over an earlier update that is still being prepared after some time. Without it, that case fails with
publication_preparation_in_progress
.
--yes
boolean
false
Confirm without a prompt. Required with
--output json
or without a terminal.
Also takes the
global flags
.
Examples
obss
experiments
publications
update
pub_2b8e5f17
--lock-version
3
--yes
--output
json
obss experiments publications unpublish
Stop sharing a public link.
obss
experiments
publications
unpublish
<
publication-i
d
>
--lock-version
<
n
>
[flags]
Plugin (MCP) tool:
unpublish_experiment
The public link stops working immediately for everyone who has it.
Arguments
Argument
Description
<publication-id>
Required. The
pub_
public link ID.
Flags
Flag
Type
Default
Description
--lock-version
integer
Required. The
lockVersion
from
obss experiments publications get
. When the link changed since, the command fails with
publication_changed
.
--yes
boolean
false
Confirm without a prompt. Required with
--output json
or without a terminal.
Also takes the
global flags
.
Examples
obss
experiments
publications
unpublish
pub_2b8e5f17
--lock-version
4
--yes
Result fields
Field
Description
status
revoked
.
publication
The public link, as
obss experiments publications get
returns it.
Notes
A project with public links can’t be deleted until you unpublish them.
obss gallery get
Read results from a public link, one section at a time. You do not need to sign in.
obss
gallery
get
<
url-or-public-i
d
>
[flags]
Plugin (MCP) tool:
get_public_experiment
Arguments
Argument
Description
<url-or-public-id>
Required. The public link URL, or its
pub_
ID.
Flags
Flag
Type
Default
Description
--section
See the table below
overview
The section to read.
--run-key
string
The run to read, for the run sections.
--trial-key
string
The trial to read, for the run sections.
--task-key
string
The task to read, for
task-details
.
--treatment-key
string
The treatment to read, for
treatment-details
.
--evaluation-key
string
The Evals attempt to read.
--insights-key
string
The Insights attempt to read.
--limit
integer
50
Results per page, up to 200 (default 50).
--cursor
string
The
nextCursor
from the previous page.
--section
Keys it takes
overview
,
runs
,
evals
,
insights
None
matrix
--evaluation-key
,
--insights-key
frictions
--insights-key
trajectory
,
output
,
files
--run-key
,
--trial-key
run-evals
--run-key
,
--trial-key
,
--evaluation-key
task-details
--task-key
,
--evaluation-key
,
--insights-key
treatment-details
--treatment-key
A key that a section does not take fails with a usage error that names the keys it takes. Also takes the
global flags
.
Examples
obss
gallery
get
pub_2b8e5f17
--section
matrix
--output
json
Notes
Read the keys for other sections from
overview
,
runs
, and
matrix
, and page with
--cursor
while
hasMore
is true. A page that’s still being built, failed, or was unpublished returns its
status
instead of items. When the public page limits your reads, the error says how many seconds to wait.
A large
frictions
result arrives as numbered parts (
part.index
of
part.count
) that share one selection. Read every page, then join the parts in order by appending each part’s clusters (a repeated
clusterKey
continues the previous cluster), family clusters, and outcome badge audits.
Progress counts
obss experiments status
,
watch
, and
get
return
progress
, which counts each run in exactly one state. For each run’s exact outcome, use
obss runs list
.
Count
Runs
queued
,
running
,
cancelled
Runs in that state. A run that was blocked and then cancelled counts as
cancelled
.
passed
,
failed
,
errored
Completed or errored runs with that result
infraBlocked
Runs that completed or errored while blocked, so Evals didn’t check them (
infraBlockedCount
in
experiments list
)
inconclusive
Completed runs with no verdict, including runs whose Evals failed
finished
The sum of
passed
,
failed
,
infraBlocked
,
inconclusive
,
cancelled
, and
errored
total
Every run
settingUp
,
agentWorking
,
evaluating
,
insights
Running runs by phase: the machine is being set up, the agent is working, Evals is checking the run, or Insights is reading it. A running run with no phase yet is in none of them.
Next steps
obss runs and results
Read each run’s result, profile, and trace.
obss schedules
Launch a finished experiment again on a cadence.
Launch an experiment
Choose tasks, treatments, agents, and trials.
Share results
Publish an experiment at a public link.
catalog and resources
schedules
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform