Benchmark
Most “AI support agent” numbers are graded by the company that built the agent. So we don’t lead with a number — we lead with a real bug, and let you check our diagnosis against the fix that actually shipped.
See it on a real bug
We pulled a real bug from the ERPNext open-source project, gave BuildWright the customer-facing symptomonly — never the fix — and scored its diagnosis against the pull request that actually merged.
The escalation
“On some multi-item invoices, the item-wise tax breakup doesn’t add up to the total tax — off by a paisa, only on certain rate × quantity combinations. The header total looks right; the per-item breakdown doesn’t reconcile. Find what’s producing it, and where.”
BuildWright’s diagnosis
Item-wise tax is rounded per line, so the accumulated per-item rounding drifts from the header total — Rounding(a) + Rounding(b) ≠ Rounding(a + b). It pointed at taxes_and_totals.py, method set_item_wise_tax(), lines 540–556.
The fix that actually merged
ERPNext PR #53925 loosened the rounding threshold in erpnext/controllers/taxes_and_totals.py — the same file and the same method BuildWright named. Right file, right method, right root cause — and it never saw the answer.
This is one of a set of real ERPNext bugs we grade this way. It doesn’t nail every one — the misses are mostly fixes that landed in JavaScript, which our code index reads less well than Python — and we’ll show you the misses too, not just this one. But this is the shape of the work: root cause and location, on a bug it had never seen, checked against the code that actually shipped.
Not cherry-picked
One bug you can check is a start; the worry is that we picked it. So we also ran BuildWright’s production agent against τ²-bench, Sierra Research’s public benchmark, where a deterministic grader — no LLM judge — checks the final system state, on someone else’s tasks. Same model as the benchmark’s own reference agent, 50 tasks each in two domains:
| Domain | BuildWright | τ²’s own agent |
|---|---|---|
| Retail | 0.82 | 0.81–0.83 |
| Airline | 0.79 | 0.80 |
Resolution rate, 50 tasks per domain; the reference agent shown on its completed runs. Read the parity, not the absolute — scores shift with domain and model.
A general-purpose platform agent tied the benchmark’s own purpose-built agent, on neutral ground it wasn’t tuned for. This is the floor, deliberately: we ran the bare loop with knowledge and learning switched off, so the comparison stayed fair. τ²-bench’s tasks are retail and airline support — proof that the engine is sound, not that it knows your domain. That proof is the real bug above, and the design-partner benchmark below.
Under the hood
Diagnosis is only as good as what the agent retrieves. So we grade the knowledge layer with an independent model — OpenAI’s GPT-4o judging BuildWright’s Claude-based output, so it isn’t marking its own work — on a public engineering knowledge base:
0.84
Faithfulness
0.68
Recall
0.88
Completeness
Faithfulness: the answer stays grounded in retrieved sources, not invented. Recall: how much of the material it needed it found. Completeness: how much of the expected answer it covered. Above the targets we gate on (0.80 / 0.60 / 0.60), on a public corpus — not your data yet.
It abstains instead of guessing.
When it can’t support an answer, it says so — in one run the agent reported “no indexed code repository” rather than inventing a file path. For a tool your engineers rely on, an honest “I don’t know” beats a confident wrong answer.
And it gets better as it learns.
Once it has learned a resolution, the facts it needs surface far more reliably next time the same problem appears — across six cases recall roughly tripled (0 → 0.52, 0.12 → 0.70, 0.28 → 0.72), with four control cases confirming it doesn’t bleed into unrelated tickets.
Straight talk: that’s measured at the retrieval level — the right facts surface. Proving it also yields better final answers is the next test on our list, and we’ll publish it when it’s run, not before.
What’s next
The bug above is real, but it’s ERPNext, not your stack. The grading approach is the point: we score against the actual merged fix, not a rubric we wrote. The version that matters runs that same objective grading on yourbacklog — your real escalations, scored against how your team actually resolved them. That’s what we build with design partners, and it’s the fastest way to know if this works for you.
The real-bug bench.Real closed bugs from the ERPNext open-source project (frappe/erpnext). The agent is shown only the customer-facing symptom, never the fix; the answer key is the merged pull request’s changed files and root cause. The example above is PR #53925. Across the set, BuildWright named the file the fix touched on the majority of bugs; the misses were mostly JavaScript-only fixes, which our Python-strong code index reads less well — a known limit, not hidden.
τ²-bench.Sierra Research’s public agentic-resolution benchmark (github.com/sierra-research/tau2-bench). BuildWright’s production agent loop is under test; τ²-bench owns tool execution and deterministic scoring, and the agent never sees the gold answer. Both agents ran Claude Sonnet 4.6; the simulated customer ran Claude Haiku 4.5. 50 tasks per domain, two trials each. Full set: resolution 0.82 / 0.79, consistency (pass²) 0.72 / ~0.74, DB-state match 0.82 / ~0.80. Caveats we state plainly: the customer is LLM-simulated, so scores don’t transfer 1:1 to live; read the parity, not the absolute against other leaderboards; a shared, throttled API key inflated the reference agent’s infra-error rate.
Knowledge layer.Measured on a public ERPNext engineering corpus, graded by an independent cross-family judge (OpenAI’s GPT-4o judging Claude output). We share the harness and per-task raw reports with design partners and serious evaluators on request — they aren’t a public leaderboard, and we don’t pretend they are.