Skip to main content
These terms mean the same thing everywhere in Obss. To see how the main parts fit together, read How Obss works.

A

Agent

The AI agent that does the work in a run, such as Claude Code or Codex, with a model and an effort level. It can be built-in or a custom agent, and you choose it at launch. See Agents.

Agent behavior

A Where to fix value: the evidence points to the agent’s own behavior, not to your product or setup. See Where to fix.

Agent default

The model choice that lets a custom agent use its own default model. See Custom agents.

Agent mode

The mode of Evals or Insights in which an Obss agent reads the whole run before it answers. Obss’s evals agent checks the claims of the agent that did the run before it scores. Obss’s insights agent checks each friction against what the tools returned, and on Comparisons it shows what a failed run and a passed run did differently. Agent mode is in private beta. Email support@obss.app to enable it for your organization. See Agent mode.

AI assistant

The tool you work in, such as Codex, Claude Code, Cursor, or GitHub Copilot. In MCP terms, it’s the MCP host. It operates Obss through the Obss Plugin or the CLI. The same product can also be an agent, but the roles differ: your AI assistant launches experiments, and agents do the tasks.

All experiments

Every experiment in the project, under Analyze. See Read results.

All frictions

Every friction in the project, under Analyze, ranked by how many runs never got past it. See Frictions.

All official treatments

All official skills, All official MCPs, All official CLIs, and All official SDKs: treatments Obss keeps for you, each with the official version of every project tool of that kind. See Treatments (Toolsets).

All runs

Every run in the project, under Analyze. See Read results.

API key

A personal key from Profile → API keys for places where you can’t sign in through a browser, such as CI. The CLI reads it from OBSS_API_KEY, and it acts as you until you revoke it. Organization API keys can’t use the CLI, and the plugin signs in with OAuth. See Profile and API keys.

Ask Obss

The in-app assistant for your project, with the same capabilities as the plugin and the CLI. You give it a model under Organization settings → Obss AI features. See Ask Obss.

Ask Obss in Slack

Ask Obss inside Slack, where the Obss app answers direct messages and channel mentions. See Use Ask Obss in Slack.

Assets

The earlier name for Tools.

B

Baseline

The treatment you compare the others with, chosen under Baseline at launch. It’s usually Raw Agent, which gives no tools. See Baseline and lift.

Brief

The note under a difference on Comparisons: one sentence on what went wrong for the side that did worse, written by the Experiment insights model, and the frictions seen in its runs. It never names a cause. Each Brief has a brf_ ID. See Brief.

C

Check for updates

Compares a file or skill imported from GitHub with its current source, so you can update it. See Tools.

CLI

  1. The Obss CLI (obss), installed from Automate → Obss CLI. See Use the CLI.
  2. A CLI tool: a command-line tool you give an agent in a treatment, under Manage → Tools → CLIs. See Tools.

Compare

A side-by-side view of two or three runs across Traces, Output, and Evals. See Compare runs side by side.

Comparisons

The experiment tab that says who did better, how sure that is, and what to do next. It opens with one sentence that answers the experiment’s question and a block for each difference, then All N comparisons. Each result is a Clear difference, a Likely difference, or a No clear difference (which doesn’t mean the two are the same). You can run a Likely difference again to be sure. Each comparison has a cmp_ ID. See Comparisons.

Criterion

One item in a rubric that a run passes or fails, added with Add criterion. See Tasks and rubrics.

Custom agent

An Agent Client Protocol (ACP) agent your organization adds, which Obss installs in each run’s machine and runs like a built-in agent. It shows a Custom tag and a status: Needs values, Needs attention, Testing, Not tested, or Ready. Custom agents are in private beta. Email support@obss.app to enable them for your organization. See Custom agents.

D

Default

The effort choice that sends no effort level, so the agent uses its own default for the model. Cursor is the exception: it uses the default variant Cursor lists for the model. See Agents.

Download

The items in a run’s ⋯ menu that save recorded work: Download trajectory, Download output files, and Download changes (diff). See Run details.

E

Edit first

Opens New experiment with a finished experiment’s tasks, treatments, and agents, so you can change them before you launch. See Launch an experiment.

Effort level

How much reasoning an agent’s model uses, where the model supports it, chosen per agent at launch. Levels can include none, minimal, low, medium, high, xhigh, and max. See Agents.

Eval set

The tasks, machine, files, tools, and treatments that evaluate one product. The Obss Plugin skills build one from a product’s docs. See Build an eval set and read its results.

Evals

How Obss scores a finished run. The Evals model reads the trajectory and changed files and marks each rubric criterion Pass or Fail, and the Evals tab notes when it couldn’t read the whole run. Evals run independently of Insights. See Evals.

Evals mode

How Evals review each run against the rubric. Standard scores each run in one fast, low-cost pass with your Evals model. Agent (private beta) uses Obss’s evals agent for an in-depth review that costs more and takes a few minutes per run. See Agent mode.

Evals model

The model that scores runs for Evals, chosen under Organization settings → Obss AI features. See Obss AI features.

Experiment

One launch of tasks, treatments, agents, and trials, where each combination is one run. It saves what it launched, so later edits to a task or treatment don’t change its results. See Experiments.

Experiment insights model

The model that groups every run’s Insights findings into frictions, and writes the Brief. See Obss AI features.

F

Family

The cause of a friction: Product, Docs & skills, Machine, Agent software, Agent reasoning, or Task quality. Each friction has one. See Family.

Files

Folders you attach to a task, which Obss puts in the agent’s working directory before the run starts. Create them under Manage → Files, by upload or from GitHub. See Machines and files.

Friction

One problem that agents hit, tracked across runs and experiments with a stable frc_ ID, one family, and one place to fix it. Most come from Insights. A run that fails before the agent answers also counts toward a friction for its cause. See Frictions.

From GitHub

Imports a file or skill from a public github.com repository, which you can later compare with Check for updates. See Tools.

I

ID

A record’s stable public identifier, with a prefix for each kind: exp_ experiments, tsk_ tasks, trt_ treatments, frc_ frictions, sch_ schedules, and tag_ tags. The CLI and the plugin use these IDs. See IDs and limits.

Inactive

A file or skill no longer offered for new tasks or treatments, but kept visible so past results make sense. A GitHub import becomes inactive when Check for updates finds its source removed. See Tools.

Insights

The analysis of a finished run’s trajectory, in which the Insights model finds frictions and patterns. Insights run independently of Evals. See Insights.

Insights mode

How Insights review each run. Standard reviews each run in one fast, low-cost pass with your Insights model. Agent (private beta) uses Obss’s insights agent for an in-depth review that costs more and takes a few minutes per run. See Agent mode.

Insights model

The model that finds the frictions and patterns in each run. See Obss AI features.

Integrations

Your personal connections under Profile → Integrations: personal provider subscriptions, GitHub, Ask Obss, and MCP server sign-ins. Select Disconnect to stop using a connected account. See Profile and API keys.

L

Launch again

Launches a new experiment with a finished experiment’s tasks, treatments, agents, and settings, using each task’s latest version. CLI: obss experiments rerun. Plugin: Launch an experiment again (rerun_experiment). See Launch again.

Launch now

Launches a schedule’s experiment right away, under your account, without moving its next launch. See Schedules.

Launch plan

An experiment’s exact selection as the CLI and the plugin send it: task versions, treatments, agents, trials, and settings. Copy a finished experiment’s plan with obss experiments clone-plan or get_experiment_plan to change and relaunch it. See obss experiments launch.

Lift

The difference in pass rate between a treatment and the baseline on the same tasks and agents. See Baseline and lift.

M

Machine

The environment a run executes in, set on the task. The default is the Standard Dev Machine, and you create others under Manage → Machines (Environments). See Machines and files.

Machine and task frictions

Frictions in the Machine or Task quality family. They come from the experiment’s own machine or task, not the product you measure, so check them before you trust the results. Don’t confuse them with Setup under Where to fix. See Machine and task frictions.

Main comparisons

The comparisons that answer an experiment’s question, such as each treatment against the baseline for all agents together. Comparisons shows them first, and checks the others (the breakdowns) more strictly. See Main comparisons and breakdowns.

Matrix

An experiment’s first tab: a heat map of tasks by agent and treatment. Each cell shows the chosen metric: Evals (the pass rate), Steps, Tool calls, Treatment use, Frictions, Tokens, Cost, or Duration. See Matrix.

Max Concurrency

How many agent sessions can run at once in a launch. It doesn’t control Evals or Insights. See Launch an experiment.

MCP

  1. The Obss MCP server: how your AI assistant operates Obss, through the Obss Plugin. See the Plugin (MCP) reference.
  2. An MCP server tool: an MCP server you give an agent in a treatment, under Manage → Tools → MCP servers. See Tools.

Members

The people in your organization, who share its projects, model providers, agents, and run balance. Manage them under Organization settings → Members. See Set up your organization.

Metrics

A run’s steps, tool calls, tokens, cost, and duration: how hard the agent worked, not only whether it passed. See Read results.

Model providers

Your organization’s API keys and gateways under Organization settings → Model providers, and the personal subscriptions that each member connects under Profile → Integrations. Agents and Obss AI features use their models. See Model providers.

N

Notifications

Your in-app notifications, under the header bell. Set Email and In-app delivery per type under Profile → Notifications. See Notifications.

O

Official version

The one version of a tool that the project marks as part of its intended interface, shown with an official mark and used by the All official treatments. See Tools.

Obss AI features

Your organization’s model choices for Obss’s own work: the Evals model, Insights model, Experiment insights model, and Ask Obss model. You can add custom instructions for Evals, Insights, and Experiment insights. See Obss AI features.

Obss Plugin

The package for your AI assistant that connects the Obss MCP server and adds skills for building eval sets, launching experiments, and reading results. See Install the Obss Plugin.

Organization

Your company’s account, which holds its projects, members, model providers, and run balance. Change it under Organization settings. See Set up your organization.

Output

The run tab for what the agent delivered: its final message under Answer, and changed files with diffs under Files. See Run details.

P

Parent experiment

The experiment that another one was launched from, with Launch again, Edit first, or a schedule. Experiment lists show it as Parent: and its name, and Open parent experiment opens it. The experiments launched from it are its child experiments. See Launch an experiment.

Pass rate

The share of runs that passed the task’s rubric. Hover over a Matrix cell to see the count, such as “3 of 4 trials passed”. See Pass rate and lift.

Pause

Stops a schedule’s launches until you select Resume. Obss also pauses a schedule that needs a fix and marks it Needs attention. See Schedules.

Plan runs

The runs your free or paid plan gives your organization each billing period. Unused plan runs don’t roll over. See Runs and billing.

Project

The place that holds your tasks, files, machines, tools, treatments, experiments, and schedules. An organization can have many, and you switch between them in the header. See Projects. A link that lets anyone view an experiment’s results without an account, created with Share on a finished experiment. Updates go live only when you publish them. See Share results.

Public sharing

The Organization settings tab that lists every public experiment link in the organization. See Share results.

Publish

Creates a public experiment link, or updates it with the latest results. CLI: obss experiments publish. Plugin: Publish an experiment (publish_experiment). See Share results.

R

Raw Agent

The treatment with no tools: the agent as its vendor ships it. It’s the usual baseline. See Treatments (Toolsets).

Recovery

Whether the agent got past a friction: Recovered, Unrecovered, or Unknown. See Recovery.

Repeat

When a schedule launches: every hour, day, weekday, week, or month, or on a custom cron expression. Schedules start on the quarter hour. See Schedules.

Rerun evals

Rescores the same recorded work without rerunning the agent, such as after a rubric or Evals model change. Before Evals have run, the action is Run evals. See Run Evals again.

Rerun Insights

Analyzes the same recorded work again without rerunning the agent. Before Insights have run, the action is Run insights. See Run Insights again.

Rubric

The criteria that Evals check each run against to decide whether it passes the task. See Tasks and rubrics.

Run

One agent’s attempt at one task with one treatment, and the unit you pay for. Errored or cancelled runs don’t count against your organization’s run balance. See Runs and billing.

Run profile

The side rail on a run that shows treatment use, tool calls, what stood out, and where the time went. CLI: obss runs profile. Plugin: Get a run profile (get_run_profile). See Run profile.

Runs

  1. The Runs tab: the tab on an experiment, task, tool, treatment, machine, file, agent, or schedule that lists its runs.
  2. Run balance: the runs your organization can spend, from plan runs and top-up runs. See Runs and billing.

S

Schedule

A rule under Automate → Schedules that launches a finished experiment again on a repeat, as a new experiment that uses runs. It runs as whoever created it, resumed it, or last changed its repeat or experiment. See Schedules.

SDK

A software development kit you give an agent in a treatment, created under Manage → Tools → SDKs. See Tools.

Share

Opens the Share experiment dialog on a finished experiment, where you create a public experiment link or invite a teammate to view it. See Share results.

Skill

A folder of instructions and files, with a SKILL.md, that an agent can read when a treatment includes it. Create skills under Manage → Tools → Skills. See Tools.

Standard Dev Machine

The default machine for a task: Ubuntu 24.04 with Node.js 24, Python 3.12, and Git. See Standard Dev Machine.

Standard mode

The default mode of Evals and Insights, which reviews each run in one pass with your Evals model or Insights model. See Evals mode and Insights mode.

T

Tag

A label with a stable tag_ ID that groups tasks, files, machines, tools, treatments, or experiments, so you can filter lists with Tags. Tags don’t change a run. See Tags.

Task

The work an agent attempts: instructions, a rubric, files, and a machine. Create tasks under Manage → Tasks. See Tasks and rubrics.

Template

A ready-made experiment on Home. Selecting one adds its tasks, tools, and treatments to your project and opens New experiment. See Templates.

Test agent

Launches one short run with the Raw Agent treatment (no tools) to check that a custom agent installs, starts, and answers. It uses one run from your balance. See Custom agents.

Timeline

The top of the Traces tab, showing when tool calls happened, grouped by model turn when the run recorded turns. See Timeline.

Tools

Skills, MCP servers, CLIs, and SDKs, under Manage → Tools. You attach them to treatments, not tasks. See Tools.

Toolset

The set of tools in a treatment. See Treatment (toolset).

Top-up runs

Runs you buy on top of your plan, which never expire. See Runs and billing.

Traces

The run tab with the recorded trajectory: the Timeline, then the transcript grouped by model turn. See Run details.

Transcript

A run’s recorded trajectory, shown on Traces grouped by model turn. To save it, choose ⋯ → Download trajectory on the run. See Run details.

Treatment (toolset)

One set of tools you compare: the skills, MCP servers, CLIs, and SDKs an agent gets in a run. Raw Agent gives none. Create treatments under Manage → Treatments (Toolsets). See Treatments (Toolsets).

Treatment use

Whether the agent used the treatment’s tools: Used, Not used, Not measured, Can’t tell, or Not applicable. It shows in the Run profile and the Runs tab. See Treatment use.

Trial

One repeat of the same task, treatment, and agent, set at launch. Each trial is one run against your run balance. See Launch an experiment.

U

Unpublish

Stops sharing an experiment, so its public link no longer works. See Share results.

Upload

Adds a local folder or ZIP archive as a file or skill, or imports tasks from a ZIP archive. CLI: obss uploads commands. Plugin: Start an archive upload (create_upload). See obss uploads create.

V

Version

A saved state of a task or tool. An experiment records the versions it launched, while Launch again and schedules use each task’s latest version. See Tasks and rubrics.

W

What the agent did

One sentence on a fixable friction that says what the agent did, shown after the suggested fix. It isn’t the change to make. See Where to fix.

Where it happens most

On a friction, the agent or treatment that hits it more often than the others in the same experiments. See Where it happens most.

Where to fix

The main friction filter: Product, Custom agent, Setup, or Agent behavior. Each friction’s suggested fix names one place your team controls. See Where to fix.