# Tasks and rubrics - Obss

Define tasks
Tasks and rubrics
Write tasks for agents, define the rubric that decides pass or fail, import tasks in bulk, and know when to rerun evals.
A task is the work you ask an agent to do, plus everything that must stay the same for a fair comparison. Its rubric decides whether a run passed.
What a task holds
Tab
What it holds
Instructions
The task name and what the agent should do
Rubric
The criteria that decide pass or fail
Files
Folders placed in the agent’s working directory
Machine
The environment the run executes in
The task’s
Runs
tab lists the runs of that task version in every experiment, and its
Experiments
and
Frictions
tabs list those runs’ experiments and frictions.
Set up files and machines in
Machines and files
. Tools attach to a
treatment
, not a task, and you choose agents at launch.
Create a task
Dashboard
CLI
Plugin (MCP)
Open
Manage → Tasks
and select
New task
.
Enter a
Task name
and the
Instructions
.
Add criteria under
Rubric
.
Attach folders on
Files
, and choose a machine on
Machine
or keep the project default.
Select
Create task
.
You can also create a task from the task list in
New experiment
.
Write the task as a JSON operation and apply it:
obss
schema
resources
apply
obss
resources
apply
--operation-file
task.json
The schema lists the fields, including
instructionsMarkdown
, the rubric, and
assetAttachments
with each file’s
mountPath
.
obss resources apply
has an example
task.json
.
Ask your AI assistant to create the task. It calls
Create or update a resource
(
author_resource
).
Create an Obss task named "Fix the checkout total" that asks the agent to fix the rounding bug in helpdesk-web/, with a rubric of three yes/no criteria.
Write the instructions
Brief the agent as you would a capable engineer who is new to your product: the goal, the output you expect, and the limits. Vague tasks are hard to score, so be specific: “Refactor the auth module so
require()
calls are deterministic, without changing the public API.”
Refer to attached folders by their
path relative to the working directory
, for example “Fix the app in
helpdesk-web/
.”
Write the rubric
A rubric is a list of
criteria
, one outcome per row, and
Add criterion
adds the next row. Write each criterion so a judge can decide it for a single run:
“Returns the five prices in page order” checks the output.
“Calls the batch endpoint once instead of one request per page” checks a step.
After each run,
Evals
use a judge model to read the run’s
recorded work
, including the files it changed, and give each criterion a result. By default, criteria are
Yes / no
, and a run passes when it meets all of them.
Advanced mode
lets you choose how each criterion gets its result and what makes the run pass.
Draft a rubric with Ask Obss or your AI assistant
Ask Obss
or your AI assistant can draft the criteria for you. Tell it what “done” means, and it saves the rubric to the task. Review the rubric before you launch.
Dashboard
CLI
Plugin (MCP)
Open
Ask Obss
and ask it to draft the rubric:
Draft a rubric for the task "Fix the checkout total". Done means the total is correct for discounts and the existing tests still pass.
The CLI doesn’t draft criteria. Write the rubric in the task’s JSON operation and apply it with
obss resources apply
.
Ask your AI assistant to draft and save the rubric. It reads the task with
Get a resource and its versions
(
inspect_resource
) and saves it with
Create or update a resource
(
author_resource
).
Task versions
Each saved change to a task or its rubric makes a new version. Older versions can’t change, and every run stays tied to the version that produced it.
Launch
Task version used
New experiment
The current version, unless you pick another in the task list
Launch again
and scheduled launches
The latest version
obss experiments rerun --versions experiment
The experiment’s own versions
Rerun evals or launch a new experiment
You changed
Do this
Only the rubric
Select
Rerun evals
on the experiment (
Run evals
if Evals haven’t run) to score the same recorded work against the new rubric.
The instructions, files, or machine
Launch a new experiment. The recorded work came from the old task, so rescoring it doesn’t measure the change.
Import tasks
To add many tasks at once, import a ZIP file.
Dashboard
CLI
Plugin (MCP)
Open
Manage → Tasks
.
In
More task actions
, select
Download template
for the ZIP format.
In
More task actions
, select
Import tasks
and drop your ZIP file.
obss
uploads
create
./tasks.zip
--kind
task
obss uploads create
uploads the ZIP and imports its tasks.
Your AI assistant calls
Start an archive upload
(
create_upload
) and then
Complete an archive upload
(
complete_upload
).
From the CLI and the plugin
Task
CLI
Plugin (MCP)
List tasks
obss catalog list --kind task
inspect_catalog
Read a task and its versions
obss resources inspect task
inspect_resource
Create or change a task
obss resources apply
author_resource
Create or change up to 20 at once
obss resources bulk-apply
bulk_author_resources
Import tasks from a ZIP
obss uploads create --kind task
create_upload
,
complete_upload
Rerun evals after a rubric change
obss evals trigger
trigger_evals
Next steps
Machines and files
Give a task its starting folders and the machine it runs on.
Tools
Add the skills, MCP servers, CLIs, and SDKs you want to compare.
Launch an experiment
Run your tasks with agents and treatments.
Evals
Learn how each run is scored against the rubric.
Ask Obss in your editor
Machines and files
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform