Benchmarks
Spider 2.0-DBT (68 DuckDB and dbt tasks), ADE-bench (dbt Labs), Terminal-Bench 2.0 via Harbor, and your private tasks. Adapters read the existing formats; the layouts come from public descriptions and are unverified.
Benchmark profile
In normal use, Ogun tries to finish the task. In a benchmark run it switches that help off, so the score measures the agent under test.
A benchmark is a set of runs
Run = Workflow + Environment + Agent + Policy + Inputs + Evaluator
Agents
An agent is a command a person configures in the matrix: name, version, model, command and the model credentials to pass by name. It never comes from a prompt or a task.
Evaluators
Official graders, run unmodified, with their version recorded next to the score.
Isolation
| Setting | Default in the benchmark profile |
|---|---|
| Runtime | One docker or podman container per run, with every capability dropped; a local runtime for trusted tasks only. MicroVMs are later |
| Network | Denied unless a person allows it in the matrix, separately for the agent and the evaluator phase; a run that cannot enforce the denial is refused |
| Filesystem | /workspace read-write · /benchmark read-only · the evaluator’s /tests appear only after the agent has finished |
| Secrets | Only the model credentials a person lists, passed by name |
| Environment variables | Cleared; no OGUN_* or ZEUS_* variable reaches the agent |
| Cache mode | disabled (default) · benchmark_provided · cold · warm |
| Off limits | Ogun and Zeus stores and state, ogund, zeusd, browser sessions: ogun-bench never connects to them |
Trajectories
A trajectory is an append-only JSON Lines file. Every line carries seq, at, elapsed_ms, agent_run, type and the BLAKE3 of the previous line, so a reader stops at the first break. Text from an agent, a model or a tool lives only in members named untrusted_*, with secrets replaced.
task.started workspace.snapshot agent.started message model.request model.response tool.call tool.shell tool.result file.read file.created file.patch file.deleted agent.output usage agent.exited db.state evaluator.started evaluator.finished task.finished
The event types of O-05 section 7. ogun-bench replay verifies a trajectory and prints it. The ogun-bench binary is not in release 0.0.1a; build it from source.
Metrics
Cost per success
Cs = Σi costi / successful tasks
Compares agents at the same pass rate by what each success costs.
Intervention rate
interventions / run
How often a person or an outside recovery had to step in.
Tool efficiency
useful tool calls / all tool calls
Proxies: duplicate, failed and repeated commands; files re-read.
Edit efficiency
necessary changed lines / all changed lines
Catches agents that thrash a repository (needs task-aware heuristics).
Recovery
detection rate · steps to recovery · recovery success
Does the agent notice a failure, and how fast does it get back on track?
Harness gain
HGE(H) = (1/N) Σm [ Score(m, H) − Score(m, Hbase) ]
How much capability a harness unlocks across N models.
Tool ablation
Δi = Scorefull − Scorefull − tool i
Which affordance actually helps, one removal at a time.
Built: token and cost totals, tool-call counts and the proxies for tool efficiency of O-05 section 7 (failed calls, repeated commands, files read again). Harness gain and tool ablation are next on the roadmap.
Research questions
Hold the model fixed and vary the harness: shell only, shell plus structured database tools, MCP data tools, Ogun’s workflow and artifact tools. Harness gain averages the difference across models. Tool ablation removes one affordance at a time. Both need exactly what Ogun records anyway: controlled environments, trajectories and costs.
No results yet. The benchmark profile is built and its adapters are being checked against the official task formats. We will publish methods before numbers.
Prior art
| Project | What it is | How Ogun relates |
|---|---|---|
| Harbor | The task format and harness Terminal-Bench 2.0 ships in | Read Harbor tasks through an adapter |
| ADE-bench harness | dbt Labs’ runner for analytics and data-engineering tasks | Run its tasks; use its graders unmodified |
| Spider 2.0 framework | The agent framework published with Spider 2.0 | Run Spider2-DBT tasks with the official evaluation |
| Inspect | A general framework for evaluating language models | Complementary; export trajectories and scores |
Tell us what to support
If you build or evaluate coding and data agents, tell us which benchmarks and agents you use. That decides which adapters are built first.