Launch plan
A launch plan is one JSON object that names every task version, treatment, and agent, so the same plan always launches the same work.preflight and launch both read it, and obss schema experiments launch lists every field and limit.
To build a plan, run obss experiments clone-plan on an experiment you already ran, and edit it. For your first experiment, copy the example below and replace its IDs with the IDs from obss catalog list --projection full. For a baseline, use the trt_ ID of the project’s Raw Agent treatment.
Example
This plan runs 2 tasks × 2 treatments × 2 agents × 2 trials, which is 16 runs:Fields
A plan can hold at most 100,000 runs. Keep any extra fields that
clone-plan adds, such as a treatment’s contentHash.
obss experiments preflight
Check a launch plan without launching it or spending runs.preflight_experiment
Flags
Pass exactly one of
--plan or --plan-file. Also takes the global flags.
Examples
Notes
Preflight exits
0 whether or not the plan is ready, so a script either checks ready or passes --require-ready. A plan with evalsMode or insightsMode set to agent gets the blocker agent_mode_unavailable when your organization can’t use that Agent mode, or agent_mode_model_unsupported, with a setupUrl to Obss AI features, when a model can’t run it. A plan that doesn’t match the schema fails with a usage error (exit code 2) that names the first wrong fields, and details lists every issue.
obss experiments launch
Launch an experiment from a plan.launch_experiment
This command spends runs from your organization’s balance and doesn’t ask you to confirm, so run preflight first. In Ask Obss, you approve each call.
Flags
Pass exactly one of
--plan or --plan-file. Also takes the global flags.
Examples
Launch, then watch until the experiment ends:
Notes
obss experiments status and obss experiments watch take either ID. To retry after a lost response, send the same plan with the same launchAttemptId, and you get the first launch back. A refused launch exits with code 4, and error.code is one of these:
In a pipeline, build a new
launchAttemptId for each launch, such as from the pipeline run number, and reuse it only to retry that launch. A saved plan file keeps its old launchAttemptId and revisions, so launch the plan that preflight returns, not the file.
obss experiments status
Read an experiment’s progress once.get_experiment_status. For an adm_ launch ID: get_launch_status.
Arguments
Flags
Also takes the global flags.
Examples
Notes
When
terminal is true, read the final runs with obss runs list. If you run obss runs list while a --wait-ms status read is still waiting, it can answer first with an older state.
obss experiments watch
Wait until an experiment ends, then print its final status.get_experiment_status, get_launch_status. MCP hosts poll the status with wait_ms.
Arguments
Flags
Also takes the global flags.
Examples
terminal is true, the command returns the same fields as obss experiments status: experiment, progress, terminal, and pollAfterMs.
Notes
Each read waits in the server for a change, and the command prints a progress line on stderr after each one unless you pass --quiet. Right after a launch, it also waits the few seconds an experiment can take to appear.
On a timeout, the message is
Experiment did not reach a terminal state after <n> seconds., and error.details has the id you watched, the experimentId when known, and the last status read. The experiment keeps running; watch it again. For pass counts, run obss results analyze after the watch ends.
obss experiments list
List and filter the experiments in the project.list_experiments
Flags
The first eleven flags are the run scope flags except
--experiment-id. Also takes the global flags.
Examples
List the experiments tagged for two releases:
Unlike
obss runs list, the result has no scope.
Notes
A run scope flag lists the experiments that have at least one run in that scope: the experiments of the runs that obss runs list returns with the same flags. --schedule-id also lists the experiments that the schedule launched before they have runs. An ID that the project doesn’t have fails with scope_not_found, and an unknown tag fails with tag_not_found, with details.tags naming it.
obss experiments get
Read an experiment’s progress, where it came from, who launched it, and the runs it used.get_experiment
Arguments
Takes only the global flags.
Examples
Notes
A deleted experiment or schedule in
lineage shows as deleted.
obss experiments brief
Read an experiment’s Brief: each clear or likely difference as Comparisons shows it, with one written sentence and the frictions seen with the side that did worse.get_experiment_brief
Arguments
Flags
Also takes the global flags.
Examples
Read the Brief:
obss evals compare prints it, Brief: and the sentence, and one line per friction with the failed runs of the side that did worse and the passed runs of the other side. A Likely difference prints its next step and command on its result line, and a Clear difference with a friction to fix in the product prints After a fix, run the N tasks that differed again · N runs with its command. After the differences come the state message, if any, what the differences leave out, the link to Comparisons, and Notes.
With Insights in Agent mode, each difference ends with What the runs did differently. Each task has one line for the failed run and one for the passed run, with the step lines they quote and the run IDs. Once the insights agent has read every pair, the Brief: sentence says what the runs did across the tasks, and the command prints the example task with “Like N other tasks”, then each task that went differently with “Went differently”. Before that, it prints the first three tasks, and “Reading N more tasks…” while the agent is still reading. Last, it says how many tasks are left, with the --output json command that returns them all:
Each finding has these fields:
Each pair in
explanations has these fields:
worse and better each have these fields:
Notes
The command changes nothing. Obss chooses the differences and counts every number from the experiment’s Evals verdicts and frictions. A model writes only each
sentence and, in Agent mode, each pair’s line. A friction listed with a difference was seen with the side that did worse. That does not mean it caused the difference.
In Agent mode, a pair says where two runs went different ways, not why the difference happened. What the insights agent used to read the pairs counts in the Insights attempt’s agentUsage (obss insights list).
- The findings are the differences that Comparisons shows as blocks. A No clear difference, a result for whether a treatment helps one agent more, and one agent’s row of a difference already shown for all agents are never findings.
- A friction is listed with a difference only when at least 3 counted runs of
worseSidehave it, on at least 2 tasks, at a higher rate than the other side. See Brief. - In Agent mode, only runs that Evals marked Pass or Fail are paired, one pair per task.
- Public experiment links don’t show the Brief.
obss experiments clone-plan
Get a launch plan from an experiment, ready to edit and launch.get_experiment_plan
Arguments
Flags
Also takes the global flags.
Examples
Save a plan, change it, check it, and launch it:
Notes
The command changes nothing. Text output only counts the version changes, so use
--output json to get the plan.
--comparison takes the cmp_ ID of a Likely difference from obss evals compare, and the plan runs that likely difference again. Launching the plan uses runs from your run balance.
- Tasks: The plan uses the comparison’s tasks, except tasks that already passed in every run, or failed in every run, at least 4 times on both sides together.
- Sides: A treatment comparison keeps its agent, or every agent for all agents together. An agent comparison keeps its treatment. A comparison of whether a treatment helps one agent more (
interaction) keeps both treatments and both agents. - Runs each: Each combination gets enough runs that the new experiment will likely settle the difference: from the experiment’s own (at least 3) up to 10.
- Versions: The plan keeps the experiment’s task versions unless you pass
--versions latest. Treatments use their current tools, as with Launch again. - Evals: Evals is on, with the mode, Evals model, and instructions of the experiment’s latest Evals attempt (the one that found the difference), even when the experiment launched with Evals off.
- Name: “Run again:”, followed by the two sides and what they share, for example “Run again: Stripe CLI vs Raw Agent, Codex · gpt-5.5 · high”. For all agents together, the name ends with the two sides.
obss experiments rerun
Launch a finished experiment again in one call.rerun_experiment
This command spends runs from your organization’s balance. In Ask Obss, you approve each call.
Evals and Insights run in the modes the experiment launched with, including Agent mode. If your organization can no longer use that Agent mode, the launch is refused with agent_mode_unavailable. To launch in Standard mode, plan the launch with clone-plan and set evalsMode or insightsMode to standard.
Arguments
Flags
Also takes the global flags.
Examples
clone-plan, edit it, and launch it.
Result fields
The same fields as obss experiments launch: launchId, experimentId, status, runCount, and dashboardUrl. The text output also prints the obss experiments watch command for the launch.
obss experiments cancel
Stop the unfinished work of an experiment, or of one run.cancel_experiment
Arguments
Flags
Also takes the global flags.
Examples
obss experiments publish
Publish a completed experiment at a public link that anyone can open without an account.publish_experiment
This command makes results public. In Ask Obss, you approve each call.
Arguments
Flags
Also takes the global flags.
Examples
Notes
The page includes runs, traces, outputs, Evals, Insights, and text files up to 256 KiB. Larger files, binary files, and private machine settings aren’t shared. See Share results.
Later results go live only when you publish an update.
obss experiments publications list
List your organization’s public links.list_experiment_publications
Flags
Also takes the global flags.
Examples
obss experiments publications get
Read one public link, its status, and the version it shows.get_experiment_publication
Arguments
Takes only the global flags.
Examples
Read the
lockVersion before you update or unpublish:
obss experiments publications update
Publish the latest results to an existing public link.update_experiment_publication
This command makes results public. In Ask Obss, you approve each call.
Arguments
Flags
Also takes the global flags.
Examples
obss experiments publications unpublish
Stop sharing a public link.unpublish_experiment
Arguments
Flags
Also takes the global flags.
Examples
Notes
A project with public links can’t be deleted until you unpublish them.
obss gallery get
Read results from a public link, one section at a time. You do not need to sign in.get_public_experiment
Arguments
Flags
A key that a section does not take fails with a usage error that names the keys it takes. Also takes the global flags.
Examples
overview, runs, and matrix, and page with --cursor while hasMore is true. A page that’s still being built, failed, or was unpublished returns its status instead of items. When the public page limits your reads, the error says how many seconds to wait.
A large frictions result arrives as numbered parts (part.index of part.count) that share one selection. Read every page, then join the parts in order by appending each part’s clusters (a repeated clusterKey continues the previous cluster), family clusters, and outcome badge audits.
Progress counts
obss experiments status, watch, and get return progress, which counts each run in exactly one state. For each run’s exact outcome, use obss runs list.
