Skip to main content
All resources

By obss team. Updated . Experiment figures identify their source snapshots and review dates below.

How to test SDK usability with Claude Code and Codex

SDK usability testing checks whether a coding agent can install, authenticate, and complete a real product workflow with your SDK. Give Claude Code or Codex a customer goal in a clean project, run the generated integration, independently verify the result, and record whether the SDK actually participated. Repeat with pinned agent, package, and environment versions.

For the managed product workflow, see agent usability testing with Obss: define customer tasks, compare interface conditions, and inspect the execution evidence behind the outcome.

Obss combines SDK usability testing with agent observability. Compare current and revised SDK materials in repeatable product experiments, then inspect requirement-level verdicts, trajectories, generated files, and frictions. This guide shows how to design the test and how our public experiments distinguish a plausible integration from a verified one.

Define what a clean project contains

A clean project needs an explicit boundary. Record the language and runtime, package manager, starter files, network access, account permissions, and documentation available to the agent. Exclude private implementation hints and hidden maintainer setup unless customers receive them too.

Use two labeled tracks when necessary:

  • Onboarding: the agent must discover the installation and authentication procedure, install the SDK, and complete the task.
  • Configured integration: the correct SDK and scoped test credentials are already available; the agent must use them in an application.

The first track measures the whole first-use experience. The second isolates the integration work after setup. An authentication outage should not be mistaken for a method-naming problem.

New evidence: a controlled SDK documentation pilot

Our 32-attempt Qdrant documentation pilot compares a pinned README with the same README plus workflow notes. All 24 integration implementations pass the supplied checks on independent replay; the original model grader flags one persistence attempt. The report preserves both evidence layers, explains the vector-normalization grading discrepancy, and reports tokens and agent work time separately by configuration. Use the executable SDK starter for an independent artifact comparison.

Test a customer goal beyond the quickstart

Choose a workflow with an observable outcome. For example, ask the agent to integrate a document-processing SDK into a starter app, submit a supplied fixture, wait for processing, and save the resulting output. Keep the acceptance test outside the instructions given to the agent.

This illustrative task can expose missing initialization, invented methods, incorrect asynchronous handling, and output that looks convincing but was never produced by the product. Include an existing-project task as well as a clean installation task if customers commonly integrate into an established codebase.

Avoid writing a hidden sequence of required function calls into the rubric when multiple correct SDK paths exist. Require a specific path only when that path is the behavior you intend to test.

Verify the result and use of the SDK

Score these questions independently:

Check What would establish it?
Installation The intended package and version were installed in the test project
Authentication A real authorized operation succeeded with the scoped test account
Integration The generated application runs against the intended product
Correct outcome Returned or persisted state satisfies the task's requirements
SDK participation Execution evidence shows the required SDK path was used
Accurate reporting Claimed identifiers and results match the observed operation

An import statement alone does not establish SDK participation. Conversely, a direct API request may be a valid solution to the user's broad problem while failing an SDK-specific usability test. Decide how to report that before seeing the results.

The public RunType experiment includes a ticket-triage task that makes product participation explicit: the rubric checks for a saved flow or agent and its returned ID, five actual ticket executions, and a report consistent with those results. This illustrates how a rubric can check product use beyond plausible final text.

RunType SDK case: a saved flow did not establish a verified workflow

Our public RunType ticket-triage task asked the agent to build a flow or agent that returned a category and priority for each of five support tickets. The report had to identify the saved object and match what RunType actually returned. The published rubric checked those requirements separately.

In the SDK condition, the requirement-level results were:

Requirement met Claude Code SDK trials Codex SDK trials
Created a saved flow or agent and reported its returned ID 2 / 2 2 / 2
Ran all five tickets and reported results matching the captured returns 2 / 2 0 / 2
Used the allowed category and priority labels 2 / 2 2 / 2
Passed every requirement for the task 2 / 2 0 / 2

Snapshot v3, requirement comparison rechecked November 2026: Claude Code 2.1.283 with Opus 5; Codex 0.158.0 with gpt-5.6-sol. Effort was unspecified. These are two historical repeats per configuration, not a current SDK reliability estimate.

The second requirement combines execution and reporting fidelity. A failure does not establish that no tickets executed. In Codex run r14, trial28's evaluation, the saved-flow check passed, with a creation response and matching ID. The evaluator then reported that captured execution output was truncated, preventing verification of every returned category and priority. A separate requirement failed because ticket T-1004 was reported as normal when the task required urgent.

That gives the team two different investigations: retain complete returned results for verification, and inspect the routing logic or instructions behind the priority error. Inspect the trial's trajectory, especially the creation, execution, and final report. The evaluation does not isolate whether the truncation originated in the SDK, command output, or capture layer; it is an evidence gap to investigate, not proof of an SDK defect.

For your own SDK test, retain a structured result artifact with each fixture ID, returned business fields, and operation ID where available. Verify it independently, without logging credentials. A polished final table or successful object creation cannot substitute for complete execution evidence. The agent observability guide explains how to keep a missing-evidence diagnosis separate from a product failure.

Real results: the SDK condition varied by agent configuration

Published RunType snapshot v3, reviewed October 19, 2026, contains 260 attempts: 13 tasks, five treatments, two agent configurations, and two trials per combination. The SDK and raw-agent conditions each contain 26 attempts per configuration.

Agent configuration Raw-agent condition RunType SDK condition
Claude Code 2.1.283, Opus 5 20 passed / 26 attempts 23 passed / 26 attempts
Codex 0.158.0, gpt-5.6-sol 21 passed / 26 attempts 18 passed / 26 attempts

These are aggregate counts from the published task suite, not a dedicated clean-project SDK study or a universal interface ranking. The snapshot does not specify an effort level for these configurations. The opposing directions are a reason to inspect individual tasks and configurations before declaring an SDK treatment broadly better.

For your own SDK, run the clean-project tracks explicitly. Keep the baseline and revised condition comparable, and report counts at the workflow level so one easy task does not hide a difficult integration.

Use a matrix you can inspect

One setup could use the current SDK materials and another a revised example or SDK version. Keep permissions, starter files, product state, and budgets fixed. If the revision changes the package and its documentation together, describe it as a combined intervention rather than attributing any difference to one component.

In obss, tasks hold instructions, rubrics, files, and machines; treatments hold the SDK, skills, or other assets being compared. Agents are selected when the experiment launches. Preserve the actual harness version, model, effort configuration, and trial count in the result.

How do agents discover an SDK's API?

In our 112-trial vector-database navigation study, covering 57 Claude Code and 55 Codex trials across eight runs, neither agent searched the web or fetched external documentation. They used existing knowledge, workspace files, and runtime inspection. Claude Code used Python's help() in 59.6% of its trials; Codex used inspect.getsource() in 87.3% of its trials.

Those are observations from that baseline suite, not a claim about every agent or SDK. They make the discovery surface part of the test: record which package metadata, signatures, docstrings, examples, source files, or web documentation the agent actually encountered. Test a revised docstring where the agent reads the package; test a documentation-discovery change where it never reaches the relevant page.

Seven trials used SQLite instead of the intended vector database. If you are testing SDK use, name the required product in the task and verify execution through it. If you are studying unprompted product selection, keep the goal unbranded and report selection separately. Those experiments answer different questions.

Find the first broken transition: discovery, installation, credential loading, initialization, method choice, execution, or verification. Use recorded trajectories to identify the command, returned response, and next action. The observability workflow turns that evidence into a bounded change to test.

Reusable SDK usability test checklist

  • Customer goal: include an operation with an independently observable result, beyond generating code.
  • Starting state: pin the runtime, package version, lockfile, starter project, documentation access, permission scope, agent/model, effort, and budgets.
  • Setup track: label whether installation and authentication are part of the task or already configured.
  • Product use: verify that the intended SDK executed the required operation; record direct-API or substitute-product fallbacks separately.
  • Returned evidence: retain complete fixture-level results and identifiers outside the final assistant message. Flag truncation or missing records explicitly.
  • Outcome: run a read-back or acceptance check against the resulting state, artifact, or actual service response.
  • Comparison: change one SDK version, example, description, or documentation surface at a time; reset fixtures and repeat both conditions.
  • Report: show passes over attempts per workflow and configuration, plus retries, tokens, and defined elapsed intervals. Keep environment failures and missing evidence visible.

For example, three workflows × two agent configurations × two setups × three repeats gives 36 attempts. This is an illustrative pilot size, not evidence from a completed study or a reliability threshold. The tools guide explains what a platform must retain to make this comparison inspectable.

Make the suite useful for the next release

Retain the tasks and verification checks, then compare current and candidate SDK versions under the same agent configurations. Review known failures and a sample of passes. Treat tokens, retries, and time as supporting measures alongside successful outcomes. Keep unavailable infrastructure and invalid fixtures distinct from SDK usability failures.

Set release requirements around the importance of the workflow and the evidence you have. A small repeated suite helps catch regressions; it does not justify a precise reliability claim for all customers and agents.

Questions teams ask

How do we test whether rewritten SDK docs actually help?

Compare current and revised documentation under the same task, SDK, starting state, permissions, agent configuration, and budgets. Record whether the agent retrieved the changed material, then compare verified outcomes, product use, and recovery. If it never encountered the revision, the result does not isolate the quality of that text.

How do we catch SDK usability regressions before a release?

Run the same versioned customer tasks against current and candidate SDK conditions. Check the actual product result and SDK participation, inspect new invalid calls or workarounds, and compare resource use only alongside successful outcomes. The SDK test checklist defines the evidence to retain.

Is generated code enough to evaluate our SDK?

Only if the relevant task ends at code generation. For an integration task, run the code and check the actual product operation and result. Correct-looking imports and final explanations can coexist with a broken integration.

Should we compare Claude Code and Codex with the same prompt?

Use the same customer goal and equivalent starting conditions. Record each harness and model configuration. Compare task outcomes and failure evidence, while preserving any environment differences that affect interpretation.

What should we bring to an obss evaluation?

Bring a representative integration task, a resettable test account, the SDK materials customers receive, and a check for the correct product outcome. The agent usability and observability tools guide explains how to assess the execution and evidence workflow.

For a controlled next experiment, evaluate SKILL.md activation, task success, tokens, and latency or compare CLI, MCP, and SDK setups on matched customer tasks. Both guides include real RunType results and reusable evaluation plans.

Related resources

Book an obss demo with one workflow, its current setup, and the change you want to evaluate.