# Run Ask Obss as a custom agent - Obss

Use cases
Run Ask Obss as a custom agent
Run Ask Obss in experiments as a custom agent from a CLI download, with a personal API key as its sign-in, and measure how well it answers.
Custom agents are in private beta. Email
support@obss.app
to enable them for your organization.
To measure how well Ask Obss answers questions about your projects, add it as a
custom agent
and score its replies with tasks and rubrics, as you would for any agent.
Prerequisites
Ask Obss
works for you on the
Ask Obss
page: it has a model under
Obss AI features
, and you connected it to your projects.
The ID of the project that Ask Obss should use, from
obss projects current
or the dashboard URL (
/projects/prj_…
).
The API key acts as you in every run of this agent, including runs that a teammate launches. In those runs, Ask Obss can read and change your projects, as it can on the
Ask Obss
page, and its chats appear in your Ask Obss history.
Steps
1
Create a personal API key
Nobody can sign in through a browser during a run, so the CLI signs in with a personal API key from the
OBSS_API_KEY
environment variable. In
Profile → API keys
, create a key and copy it.
2
Find the Linux x64 release of the CLI
Each run downloads the Obss CLI, which serves Ask Obss over the Agent Client Protocol (ACP) with
obss acp ask-obss
. Runs use its Linux x64 build,
obss-linux-x64
. To find that build’s URL and SHA-256:
Open
https://obss.app/install
in a browser. The installer script names the release location of the
stable
channel.
Open the release manifest,
latest.json
, at that location.
Copy the
url
and
sha256
of the
linux-x64
entry.
The URL names one release, so runs keep that exact build, and the CLI doesn’t update itself while
OBSS_API_KEY
is set. For a newer build, enter a new
url
and
sha256
in the agent’s settings, then save.
3
Create the custom agent
In
Manage → Agents
, select
Connect custom agent
and enter:
Field
Value
Name (the page title,
Name this agent
)
Ask Obss
Install
Download
Download URL
The
url
of the
linux-x64
entry
SHA-256
The
sha256
of the
linux-x64
entry
Command
obss
Arguments
acp ask-obss --project-id <project-id>
Under
Environment variables
, add
OBSS_API_KEY
with your key as its value, then select
Connect agent
. Obss saves the download, a binary rather than an archive, as
obss
and runs it with the arguments.
4
Test the agent
Select
Test agent
. The test asks Ask Obss for its name and what it can help with, and the status updates on the page. When it answers, the status is
Ready
and
Models
lists the models it offers: turn on the ones to compare.
If the status is
Needs attention
, match the error:
Error
Fix
The key isn’t valid, or you revoked it
Create a new key and set it as
OBSS_API_KEY
.
Ask Obss isn’t connected for you
Connect it on the page the error names, then test again.
The download doesn’t match its SHA-256
Copy the
url
and
sha256
from the same
linux-x64
entry.
This version of the CLI can’t run this command
The pinned build is
too old
. Enter a current
linux-x64
url
and
sha256
in
Download URL
and
SHA-256
, save, and test again.
5
Write tasks about your projects
Ask Obss calls its tools through your Obss account, not on the run’s machine, so it can’t use the run’s files, services, or a treatment’s tools. Write
tasks
that ask about your projects instead, such as “How many tasks does this project have?”, with a rubric that judges the reply.
6
Run an experiment
Select
New experiment
, and choose your tasks,
Ask Obss
with the models to compare, and the
Raw Agent
treatment (no tools), because a treatment’s tools don’t reach Ask Obss. Then select
Launch
.
Interpret the results
On the experiment’s
Matrix
with the
Evals
metric, each cell shows the share of trials in which Ask Obss’s reply passed the rubric, for each task and model. Switch the metric to
Cost
,
Tokens
, or
Duration
to compare what each model costs.
What you see
What to do
A model passes most tasks
Compare its cost and duration with the other models before you choose it for Ask Obss under
Obss AI features
.
A task fails for every model
Read a run’s
Traces
to see which Obss tools Ask Obss called, and check the rubric against the correct answer.
Runs fail before Ask Obss answers
Read the run’s
frictions
. A failed custom agent step shows under
Custom agent
in
Where to fix
.
From the CLI and the plugin
Only the agent’s API key stays in the dashboard. To create the same agent with the CLI, run:
obss
project
ai
apply
--operation
'{"action":"create_custom_agent","name":"Ask Obss","source":{"kind":"manual","install":{"kind":"download","url":"<linux-x64 url>","sha256":"<linux-x64 sha256>","command":"obss"},"args":"acp ask-obss --project-id <project-id>"},"env":["OBSS_API_KEY"]}'
--output
json
The result gives the agent key and the dashboard link where you set
OBSS_API_KEY
. Then test the agent:
obss
project
ai
test-agent
<
agent-ke
y
>
See
obss project ai apply
and
obss project ai test-agent
. From the plugin, send the same operation to
Create custom agents and change agents, models, and Obss AI settings
(
manage_project_ai
), then call
Test a custom agent
(
test_custom_agent
).
Other ways to run it
Run your own ACP agent.
Follow the same steps with your agent’s npm package, Python package, download, or ACP Registry entry. See
Custom agents
.
Track the answers over time.
Schedule the experiment to see whether the answers change as your projects do. See
Track results over time
.
Next steps
Custom agents
Connect and test your own ACP agent.
Ask Obss
Choose Ask Obss’s model and connect your projects.
Use an API key
Sign in to the CLI with a personal API key.
Find and fix frictions
FAQ
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform