# FAQ - Obss

Help
FAQ
Find answers about plans and runs, agents and models, results, sharing, Ask Obss, the CLI, and the plugin.
Can’t find your question? Email
support@obss.app
, or see
Troubleshooting
for a specific problem.
About Obss
What is Obss for?
It measures how well AI agents, such as Claude Code, Codex, and Cursor, do real work with your product, and what helps them. You build evals and custom benchmarks, compare agents and models on the same tasks, and find where agents fail. See
What is Obss
.
Why run an experiment instead of trying an agent on my own machine?
A local try runs the agent once, so it can’t tell you how often the agent succeeds. An experiment repeats the same tasks across agents, models, and treatments on fixed machines and files, records every trajectory, and marks each rubric criterion Pass or Fail. See
How Obss works
.
What can I evaluate?
Skills, MCP servers, CLIs, and SDKs, and the docs, APIs, and files they use. Put the tools in a treatment and compare it with
Raw Agent
, the usual baseline, which gives no tools. See
Measure a tool’s impact
.
Where do files, machines, and tools attach?
Files and a machine attach to a
task
, tools attach to a
treatment
(toolset), and you choose agents at launch. See
Where files, machines, and tools attach
.
Plans and runs
Is there a free plan?
Yes, with no credit card required. It resets your organization’s runs each month, and unused plan runs don’t roll over. See
Pricing
and
Runs and billing
.
How do I get more runs?
Subscribe for more runs each month from
Organization settings → Usage & billing
, or top up at any time for runs that never expire. Larger top-ups cost less per run. For volume discounts, email
support@obss.app
.
How many runs does an experiment use?
One for each combination of task, treatment, agent, and trial. For example, 3 tasks × 2 treatments × 1 agent × 2 trials uses 12 runs, and the
New experiment
footer shows
Uses 12 runs
. See
Launch an experiment
.
Do failed runs count?
A run that finishes and fails its rubric counts. Errored or cancelled runs don’t count against your organization’s run balance.
What does a run include?
The cost of running the experiment on Obss, including the cloud machines. Your providers bill model use separately, through your own API keys or subscriptions.
Does every plan include all features?
Yes. Every plan includes unlimited members, projects, experiments, tasks, treatments, and tools, and full access to the dashboard, the CLI, and the plugin.
Can I invite my team?
Yes, at no extra cost. Members share your organization’s run balance. See
Invite teammates
.
Agents and models
Which agents can I run?
Built-in agents such as Claude Code, Codex, Cursor, Antigravity, and Qwen Code. Each works only with certain providers:
Which providers each agent uses
lists them, and each agent’s page under
Manage → Agents
shows which of your providers it can use.
Custom agents are in private beta. Email
support@obss.app
to enable them for your organization, and then you can add any Agent Client Protocol (ACP) agent. See
Custom agents
.
Can I run Claude Code with Claude Pro or Max?
Yes. Claude Code can use
Anthropic (Claude Subscription)
, which you connect under
Profile → Integrations
. See
Which providers each agent uses
.
Can I use Claude Subscription for Obss AI features?
Yes, for Evals and Insights: connect
Anthropic (Claude Subscription)
under
Profile → Integrations
, then choose a Claude model under
Obss AI features
. Ask Obss and Agent mode can’t use
Anthropic (Claude Subscription)
models. See
Obss AI features
.
Which model providers can I connect?
Organization API keys, gateways such as OpenRouter and Vercel AI Gateway, and OpenAI- or Anthropic-compatible endpoints. Each person can also connect personal subscriptions under
Profile → Integrations
. See
Model providers
.
Can I compare models and effort levels?
Yes. Choose more than one model or effort level for an agent at launch, and each choice gets its own
Matrix
column. See
Compare agents and models
.
What is Max Concurrency?
The limit on how many agent sessions run at once in a launch. It doesn’t control Evals or Insights. See
Set Max Concurrency
.
Evals, Insights, and results
What is the difference between Evals and Insights?
Evals
score each run against your rubric, and
Insights
read the trajectory to find frictions. They run independently.
Can I set my own pass or fail criteria?
Yes. Write a rubric of plain-language criteria, and Evals mark each one Pass or Fail. See
Tasks and rubrics
.
Do Evals read every step?
Not always. Evals and Insights read as much of the trajectory as fits the model’s context window (Evals also read the changed files), and the result says when something was left out. On older runs, a missing note doesn’t mean that the whole trajectory was read. See
What Evals read
and
What Insights read
.
What is Agent mode?
A mode of Evals and Insights in which an Obss agent reads the whole run before it answers. For Evals, the agent checks the claims of the agent that did the run before it scores. For Insights, it checks each friction against what the tools returned, and on
Comparisons
it shows what a failed run and a passed run did differently. Agent mode costs more and takes a few minutes per run. It is in private beta. Email
support@obss.app
to enable it for your organization. See
Agent mode
.
Can I change the rubric after a run?
Yes. Edit the rubric and select
Rerun evals
on the experiment to rescore the same recorded work. If the task itself changed, launch a new experiment. See
Run Evals again
.
How do I read lift?
On
Matrix
, with the
Evals
metric, compare each treatment’s cells with the baseline’s (chosen at launch) for the same agent and task. See
Pass rate and lift
, and check whether a difference holds up on
Comparisons
.
What is the Brief?
A short note under each difference on
Comparisons
: one sentence on what went wrong for the side that did worse, and the frictions seen in its runs. Obss counts every number, and the
Experiment insights model
writes only the sentence. It needs Insights, and it never names a cause. See
Brief
.
Can I see the full trace behind a result?
Yes. Open a run:
Traces
shows the
Timeline
and transcript,
Output
shows the final answer and changed files,
Evals
shows each criterion’s verdict, and the
Run profile
rail shows treatment use and where the time went. See
Run details
.
Can I compare runs side by side?
Yes. On the
Runs
tab, select two or three runs and select
Compare
. See
Compare runs side by side
.
Can I export my results?
Yes. A run’s
⋯
menu saves the trajectory, the output files, or the changes as a diff. The CLI and the plugin read runs, run profiles, trajectories, Evals, Insights, and frictions, and
obss data query
returns rows for your own analysis. See
Run details
.
Can I share results with people outside my organization?
Yes. Anyone with a public experiment link can view the results without an account. For a teammate, invite them to the organization instead. See
Share results
.
Experiments
Can I start from a template?
Yes.
Home
has templates such as
Getting started
,
Stripe interface evals
,
Latest model shootout
, and
Skill lift
. See
Templates
.
Can I run an experiment again?
Yes. On a finished experiment, select
Launch again
to run the same tasks, treatments, and agents, or
Edit first
to change something before you launch. See
Launch again
.
Can I run experiments on a schedule?
Yes. Select
Schedule
on a finished experiment, or open
Automate → Schedules
, and set the repeat and time zone. Each launch is a new experiment that uses runs. See
Schedules
.
Can I cancel a running experiment?
Yes. Select
Cancel experiment
while it runs. Unfinished runs stop, completed runs stay available, and cancelled runs don’t count against your run balance. See
Cancel an experiment
.
Can I import skills or files from GitHub?
Yes, from public
github.com
repositories, but not private ones. Use
From GitHub
to import, and
Check for updates
later. See
Tools
.
Can I organize tasks, tools, and experiments?
Yes. Tag tasks, files, machines, tools, treatments, and experiments, then filter any list with
Tags
. Tags don’t change a run. See
Tags
.
Ask Obss, the plugin, and the CLI
What is Ask Obss?
The in-app assistant for your project: it can create tasks, launch experiments, and explain results. Give it a model from
OpenAI (ChatGPT Subscription)
,
Nous Portal
, or an organization provider under
Organization settings → Obss AI features
. It can’t use
Anthropic (Claude Subscription)
. See
Ask Obss
.
Can my AI assistant operate Obss?
Yes. With the Obss Plugin, your AI assistant (such as Codex, Claude Code, Cursor, or GitHub Copilot) can create tasks and treatments, launch and watch experiments, and read results. Launches use runs, so
turn on tool approval
in your AI assistant’s settings to have it ask you first. See
Install the Obss Plugin
.
Why can't I enter an API key from the CLI or the plugin?
You enter secrets, such as OAuth sign-ins, API keys, and secret headers, only in the dashboard. The CLI, the plugin, and Ask Obss give you a link to the exact page instead. To list every link, run
obss setup links
or ask for
Get secure setup links
(
get_setup_links
).
Can I use Obss in CI/CD?
Yes: launch and watch experiments from the
CLI
in your own automation, signed in with a personal API key in
OBSS_API_KEY
. There is no built-in CI pipeline gate. See
Run an experiment in CI
.
Where do I manage projects and account settings?
Switch
projects
in the header, use
Organization settings
for
shared setup
, and use
Profile
for
integrations, API keys, and notifications
. The
Dashboard map
shows where everything is.
Related
Troubleshooting
Fix a specific problem.
Quickstart
Launch your first experiment.
Runs and billing
See what a run is and how to get more.
Glossary
Look up a product term.
Ask Obss as a custom agent
Troubleshooting
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform