# Glossary - Obss

Get started
Glossary
Plain-language definitions of every Obss product term, with a link to the page that covers each one.
These terms mean the same thing everywhere in Obss. To see how the main parts fit together, read
How Obss works
.
A
Agent
The AI agent that does the work in a run, such as Claude Code or Codex, with a model and an effort level. It can be built-in or a
custom agent
, and you choose it at launch. See
Agents
.
Agent behavior
A
Where to fix
value: the evidence points to the agent’s own behavior, not to your product or setup. See
Where to fix
.
Agent default
The model choice that lets a custom agent use its own default model. See
Custom agents
.
Agent mode
The mode of Evals or Insights in which an Obss agent reads the whole run before it answers. Obss’s evals agent checks the claims of the agent that did the run before it scores. Obss’s insights agent checks each friction against what the tools returned, and on
Comparisons
it shows what a failed run and a passed run did differently. Agent mode is in private beta. Email
support@obss.app
to enable it for your organization. See
Agent mode
.
AI assistant
The tool you work in, such as Codex, Claude Code, Cursor, or GitHub Copilot. In MCP terms, it’s the MCP host. It operates Obss through the
Obss Plugin
or the CLI. The same product can also be an
agent
, but the roles differ: your AI assistant launches experiments, and agents do the tasks.
All experiments
Every experiment in the project, under
Analyze
. See
Read results
.
All frictions
Every friction in the project, under
Analyze
, ranked by how many runs never got past it. See
Frictions
.
All official treatments
All official skills
,
All official MCPs
,
All official CLIs
, and
All official SDKs
: treatments Obss keeps for you, each with the
official version
of every project tool of that kind. See
Treatments (Toolsets)
.
All runs
Every run in the project, under
Analyze
. See
Read results
.
API key
A personal key from
Profile → API keys
for places where you can’t sign in through a browser, such as CI. The CLI reads it from
OBSS_API_KEY
, and it acts as you until you revoke it. Organization API keys can’t use the CLI, and the plugin signs in with OAuth. See
Profile and API keys
.
Ask Obss
The in-app assistant for your project, with the same capabilities as the plugin and the CLI. You give it a model under
Organization settings → Obss AI features
. See
Ask Obss
.
Ask Obss in Slack
Ask Obss inside Slack, where the
Obss
app answers direct messages and channel mentions. See
Use Ask Obss in Slack
.
Assets
The earlier name for
Tools
.
B
Baseline
The treatment you compare the others with, chosen under
Baseline
at launch. It’s usually
Raw Agent
, which gives no tools. See
Baseline and lift
.
Brief
The note under a difference on
Comparisons
: one sentence on what went wrong for the side that did worse, written by the
Experiment insights model
, and the frictions seen in its runs. It never names a cause. Each Brief has a
brf_
ID. See
Brief
.
C
Check for updates
Compares a file or skill imported from GitHub with its current source, so you can update it. See
Tools
.
CLI
The Obss CLI
(
obss
), installed from
Automate → Obss CLI
. See
Use the CLI
.
A CLI tool
: a command-line tool you give an agent in a treatment, under
Manage → Tools → CLIs
. See
Tools
.
Compare
A side-by-side view of two or three runs across
Traces
,
Output
, and
Evals
. See
Compare runs side by side
.
Comparisons
The experiment tab that says who did better, how sure that is, and what to do next. It opens with one sentence that answers the experiment’s question and a block for each difference, then
All N comparisons
. Each result is a
Clear difference
, a
Likely difference
, or a
No clear difference
(which doesn’t mean the two are the same). You can
run a
Likely difference
again
to be sure. Each comparison has a
cmp_
ID. See
Comparisons
.
Criterion
One item in a rubric that a run passes or fails, added with
Add criterion
. See
Tasks and rubrics
.
Custom agent
An Agent Client Protocol (ACP) agent your organization adds, which Obss installs in each run’s machine and runs like a built-in agent. It shows a
Custom
tag and a status:
Needs values
,
Needs attention
,
Testing
,
Not tested
, or
Ready
. Custom agents are in private beta. Email
support@obss.app
to enable them for your organization. See
Custom agents
.
D
Default
The effort choice that sends no effort level, so the agent uses its own default for the model. Cursor is the exception: it uses the default variant Cursor lists for the model. See
Agents
.
Download
The items in a run’s
⋯
menu that save recorded work:
Download trajectory
,
Download output files
, and
Download changes (diff)
. See
Run details
.
E
Edit first
Opens
New experiment
with a finished experiment’s tasks, treatments, and agents, so you can change them before you launch. See
Launch an experiment
.
Effort level
How much reasoning an agent’s model uses, where the model supports it, chosen per agent at launch. Levels can include none, minimal, low, medium, high, xhigh, and max. See
Agents
.
Eval set
The tasks, machine, files, tools, and treatments that evaluate one product. The
Obss Plugin
skills build one from a product’s docs. See
Build an eval set and read its results
.
Evals
How Obss scores a finished run. The
Evals model
reads the trajectory and changed files and marks each rubric criterion Pass or Fail, and the
Evals
tab notes when it couldn’t read the whole run. Evals run independently of Insights. See
Evals
.
Evals mode
How Evals review each run against the rubric.
Standard
scores each run in one fast, low-cost pass with your Evals model.
Agent
(private beta) uses Obss’s evals agent for an in-depth review that costs more and takes a few minutes per run. See
Agent mode
.
Evals model
The model that scores runs for Evals, chosen under
Organization settings → Obss AI features
. See
Obss AI features
.
Experiment
One launch of tasks, treatments, agents, and trials, where each combination is one run. It saves what it launched, so later edits to a task or treatment don’t change its results. See
Experiments
.
Experiment insights model
The model that groups every run’s Insights findings into frictions, and writes the
Brief
. See
Obss AI features
.
F
Family
The cause of a friction:
Product
,
Docs & skills
,
Machine
,
Agent software
,
Agent reasoning
, or
Task quality
. Each friction has one. See
Family
.
Files
Folders you attach to a task, which Obss puts in the agent’s working directory before the run starts. Create them under
Manage → Files
, by upload or from GitHub. See
Machines and files
.
Friction
One problem that agents hit, tracked across runs and experiments with a stable
frc_
ID, one
family
, and one place to fix it. Most come from Insights. A run that fails before the agent answers also counts toward a friction for its cause. See
Frictions
.
From GitHub
Imports a file or skill from a public
github.com
repository, which you can later compare with
Check for updates
. See
Tools
.
I
ID
A record’s stable public identifier, with a prefix for each kind:
exp_
experiments,
tsk_
tasks,
trt_
treatments,
frc_
frictions,
sch_
schedules, and
tag_
tags. The CLI and the plugin use these IDs. See
IDs and limits
.
Inactive
A file or skill no longer offered for new tasks or treatments, but kept visible so past results make sense. A GitHub import becomes inactive when
Check for updates
finds its source removed. See
Tools
.
Insights
The analysis of a finished run’s trajectory, in which the
Insights model
finds frictions and patterns. Insights run independently of Evals. See
Insights
.
Insights mode
How Insights review each run.
Standard
reviews each run in one fast, low-cost pass with your Insights model.
Agent
(private beta) uses Obss’s insights agent for an in-depth review that costs more and takes a few minutes per run. See
Agent mode
.
Insights model
The model that finds the frictions and patterns in each run. See
Obss AI features
.
Integrations
Your personal connections under
Profile → Integrations
: personal provider subscriptions, GitHub, Ask Obss, and MCP server sign-ins. Select
Disconnect
to stop using a connected account. See
Profile and API keys
.
L
Launch again
Launches a new experiment with a finished experiment’s tasks, treatments, agents, and settings, using each task’s latest version. CLI:
obss experiments rerun
. Plugin:
Launch an experiment again
(
rerun_experiment
). See
Launch again
.
Launch now
Launches a schedule’s experiment right away, under your account, without moving its next launch. See
Schedules
.
Launch plan
An experiment’s exact selection as the CLI and the plugin send it: task versions, treatments, agents, trials, and settings. Copy a finished experiment’s plan with
obss experiments clone-plan
or
get_experiment_plan
to change and relaunch it. See
obss experiments launch
.
Lift
The difference in pass rate between a treatment and the baseline on the same tasks and agents. See
Baseline and lift
.
M
Machine
The environment a run executes in, set on the task. The default is the
Standard Dev Machine
, and you create others under
Manage → Machines (Environments)
. See
Machines and files
.
Machine and task frictions
Frictions in the
Machine
or
Task quality
family. They come from the experiment’s own machine or task, not the product you measure, so check them before you trust the results. Don’t confuse them with
Setup
under
Where to fix
. See
Machine and task frictions
.
Main comparisons
The comparisons that answer an experiment’s question, such as each treatment against the baseline for all agents together.
Comparisons
shows them first, and checks the others (the breakdowns) more strictly. See
Main comparisons and breakdowns
.
Matrix
An experiment’s first tab: a heat map of tasks by agent and treatment. Each cell shows the chosen metric:
Evals
(the pass rate),
Steps
,
Tool calls
,
Treatment use
,
Frictions
,
Tokens
,
Cost
, or
Duration
. See
Matrix
.
Max Concurrency
How many agent sessions can run at once in a launch. It doesn’t control Evals or Insights. See
Launch an experiment
.
MCP
The Obss MCP server
: how your
AI assistant
operates Obss, through the
Obss Plugin
. See the
Plugin (MCP) reference
.
An MCP server tool
: an MCP server you give an agent in a treatment, under
Manage → Tools → MCP servers
. See
Tools
.
Members
The people in your organization, who share its projects, model providers, agents, and run balance. Manage them under
Organization settings → Members
. See
Set up your organization
.
Metrics
A run’s steps, tool calls, tokens, cost, and duration: how hard the agent worked, not only whether it passed. See
Read results
.
Model providers
Your organization’s API keys and gateways under
Organization settings → Model providers
, and the personal subscriptions that each member connects under
Profile → Integrations
. Agents and Obss AI features use their models. See
Model providers
.
N
Notifications
Your in-app notifications, under the header bell. Set Email and In-app delivery per type under
Profile → Notifications
. See
Notifications
.
O
Official version
The one version of a tool that the project marks as part of its intended interface, shown with an official mark and used by the
All official
treatments. See
Tools
.
Obss AI features
Your organization’s model choices for Obss’s own work: the
Evals model
,
Insights model
,
Experiment insights model
, and
Ask Obss model
. You can add custom instructions for Evals, Insights, and Experiment insights. See
Obss AI features
.
Obss Plugin
The package for your
AI assistant
that connects the Obss MCP server and adds skills for building
eval sets
, launching experiments, and reading results. See
Install the Obss Plugin
.
Organization
Your company’s account, which holds its projects, members, model providers, and run balance. Change it under
Organization settings
. See
Set up your organization
.
Output
The run tab for what the agent delivered: its final message under
Answer
, and changed files with diffs under
Files
. See
Run details
.
P
Parent experiment
The experiment that another one was launched from, with
Launch again
,
Edit first
, or a schedule. Experiment lists show it as
Parent:
and its name, and
Open parent experiment
opens it. The experiments launched from it are its child experiments. See
Launch an experiment
.
Pass rate
The share of runs that passed the task’s rubric. Hover over a
Matrix
cell to see the count, such as “3 of 4 trials passed”. See
Pass rate and lift
.
Pause
Stops a schedule’s launches until you select
Resume
. Obss also pauses a schedule that needs a fix and marks it
Needs attention
. See
Schedules
.
Plan runs
The runs your free or paid plan gives your organization each billing period. Unused plan runs don’t roll over. See
Runs and billing
.
Project
The place that holds your tasks, files, machines, tools, treatments, experiments, and schedules. An organization can have many, and you switch between them in the header. See
Projects
.
Public experiment link
A link that lets anyone view an experiment’s results without an account, created with
Share
on a finished experiment. Updates go live only when you publish them. See
Share results
.
Public sharing
The
Organization settings
tab that lists every public experiment link in the organization. See
Share results
.
Publish
Creates a public experiment link, or updates it with the latest results. CLI:
obss experiments publish
. Plugin:
Publish an experiment
(
publish_experiment
). See
Share results
.
R
Raw Agent
The treatment with no tools: the agent as its vendor ships it. It’s the usual baseline. See
Treatments (Toolsets)
.
Recovery
Whether the agent got past a friction:
Recovered
,
Unrecovered
, or
Unknown
. See
Recovery
.
Repeat
When a schedule launches: every hour, day, weekday, week, or month, or on a custom cron expression. Schedules start on the quarter hour. See
Schedules
.
Rerun evals
Rescores the same recorded work without rerunning the agent, such as after a rubric or Evals model change. Before Evals have run, the action is
Run evals
. See
Run Evals again
.
Rerun Insights
Analyzes the same recorded work again without rerunning the agent. Before Insights have run, the action is
Run insights
. See
Run Insights again
.
Rubric
The criteria that Evals check each run against to decide whether it passes the task. See
Tasks and rubrics
.
Run
One agent’s attempt at one task with one treatment, and the unit you pay for. Errored or cancelled runs don’t count against your organization’s run balance. See
Runs and billing
.
Run profile
The side rail on a run that shows treatment use, tool calls, what stood out, and where the time went. CLI:
obss runs profile
. Plugin:
Get a run profile
(
get_run_profile
). See
Run profile
.
Runs
The Runs tab
: the tab on an experiment, task, tool, treatment, machine, file, agent, or schedule that lists its runs.
Run balance
: the runs your organization can spend, from plan runs and top-up runs. See
Runs and billing
.
S
Schedule
A rule under
Automate → Schedules
that launches a finished experiment again on a
repeat
, as a new experiment that uses runs. It runs as whoever created it, resumed it, or last changed its repeat or experiment. See
Schedules
.
SDK
A software development kit you give an agent in a treatment, created under
Manage → Tools → SDKs
. See
Tools
.
Share
Opens the
Share experiment
dialog on a finished experiment, where you create a public experiment link or invite a teammate to view it. See
Share results
.
Skill
A folder of instructions and files, with a
SKILL.md
, that an agent can read when a treatment includes it. Create skills under
Manage → Tools → Skills
. See
Tools
.
Standard Dev Machine
The default machine for a task: Ubuntu 24.04 with Node.js 24, Python 3.12, and Git. See
Standard Dev Machine
.
Standard mode
The default mode of Evals and Insights, which reviews each run in one pass with your Evals model or Insights model. See
Evals mode
and
Insights mode
.
T
Tag
A label with a stable
tag_
ID that groups tasks, files, machines, tools, treatments, or experiments, so you can filter lists with
Tags
. Tags don’t change a run. See
Tags
.
Task
The work an agent attempts: instructions, a rubric, files, and a machine. Create tasks under
Manage → Tasks
. See
Tasks and rubrics
.
Template
A ready-made experiment on
Home
. Selecting one adds its tasks, tools, and treatments to your project and opens
New experiment
. See
Templates
.
Test agent
Launches one short run with the
Raw Agent
treatment (no tools) to check that a custom agent installs, starts, and answers. It uses one run from your balance. See
Custom agents
.
Timeline
The top of the
Traces
tab, showing when tool calls happened, grouped by model turn when the run recorded turns. See
Timeline
.
Tools
Skills, MCP servers, CLIs, and SDKs, under
Manage → Tools
. You attach them to treatments, not tasks. See
Tools
.
Toolset
The set of tools in a treatment. See
Treatment (toolset)
.
Top-up runs
Runs you buy on top of your plan, which never expire. See
Runs and billing
.
Traces
The run tab with the recorded trajectory: the
Timeline
, then the transcript grouped by model turn. See
Run details
.
Transcript
A run’s recorded trajectory, shown on
Traces
grouped by model turn. To save it, choose
⋯ → Download trajectory
on the run. See
Run details
.
Treatment (toolset)
One set of tools you compare: the skills, MCP servers, CLIs, and SDKs an agent gets in a run.
Raw Agent
gives none. Create treatments under
Manage → Treatments (Toolsets)
. See
Treatments (Toolsets)
.
Treatment use
Whether the agent used the treatment’s tools:
Used
,
Not used
,
Not measured
,
Can’t tell
, or
Not applicable
. It shows in the
Run profile
and the
Runs
tab. See
Treatment use
.
Trial
One repeat of the same task, treatment, and agent, set at launch. Each trial is one run against your run balance. See
Launch an experiment
.
U
Unpublish
Stops sharing an experiment, so its public link no longer works. See
Share results
.
Upload
Adds a local folder or ZIP archive as a file or skill, or imports tasks from a ZIP archive. CLI:
obss uploads
commands. Plugin:
Start an archive upload
(
create_upload
). See
obss uploads create
.
V
Version
A saved state of a task or tool. An experiment records the versions it launched, while
Launch again
and schedules use each task’s latest version. See
Tasks and rubrics
.
W
What the agent did
One sentence on a fixable friction that says what the agent did, shown after the suggested fix. It isn’t the change to make. See
Where to fix
.
Where it happens most
On a friction, the agent or treatment that hits it more often than the others in the same experiments. See
Where it happens most
.
Where to fix
The main friction filter:
Product
,
Custom agent
,
Setup
, or
Agent behavior
. Each friction’s suggested fix names one place your team controls. See
Where to fix
.
Related
How Obss works
See how the parts fit together.
IDs and limits
Look up ID prefixes and limits.
FAQ
Find answers to common questions.
Quickstart
Launch your first experiment.
Dashboard map
Organization
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform