Terra and Sonnet tie on the hard battery — and fail in opposite directions

Scoreboard

Highlighted row: subject model. Writing W-L-T is the row model's wins, losses and ties against the subject, one decision per judge per task (A win requires the judge to pick the same response in both A/B orders; a split verdict is a tie.). This battery is a single agentic hard track — no b1, code, ops or context objective tracks and no writing judge were run — so the objective column is n/a and the hard-battery column is the headline. Partial credit is the mean hidden-test fraction per trial and is the tiebreak metric.

Every task, every model

Cells show passes / valid trials; colour runs red (0%) to green (100%).

What was run

The battery is eight agentic engineering tasks — hx_api_migration, hx_fleet_incident, hx_lang_interp, hx_minidb_mvcc, hx_perf_engine, hx_policy_reconcile, hx_rostering, hx_vuln_audit — each run three times at high reasoning, scored by hidden checks the model never sees. Every cell below was re-read from the run's own artifacts rather than from a summary: the per-trial rows, the run report, and the harness session store.

Both runs are frozen to the same task set, and the task identities match by hash, not by name. For all eight tasks the task content hash recorded in each run is byte-identical across the two runs — hx_api_migration fdd9a861…, hx_fleet_incident 354e0095…, hx_lang_interp 2a9c6e7d…, hx_minidb_mvcc 2f45856f…, hx_perf_engine e9f25854…, hx_policy_reconcile 93f78410…, hx_rostering 3725095a…, hx_vuln_audit 8ddec085…. Both runs also record the same scoring rules — selection, attribution, coverage, partial-credit and reliability rules compare equal field by field — so the two columns are scored the same way.

Access routes differ, and that matters for reading the numbers. Terra ran on an OpenAI subscription route: all 24 of its sessions record provider openai-codex, base URL https://chatgpt.com/backend-api/codex, billing mode subscription_included. Sonnet ran on the metered Anthropic API: all 24 of its sessions record provider anthropic and base URL https://api.anthropic.com. These are read back from each session's own record, not taken from the label on the command line; a cell answered by a fallback route would have been discarded as invalid by the harness, and none were. All 24 trials in each run are canonical, verified and provider-attributed.

The tie

Two different models arrived at the same pass count by different routes. Sonnet reached five tasks at least once and Terra four, so Sonnet's ceiling per task is a little higher; Terra solved three tasks on every attempt and Sonnet only one, so Terra's behaviour is steadier. The partial-credit column leans Sonnet by 5.5 points. None of those differences is large enough, at 24 cells, to call a winner — and the pass column, which is the only one with a pre-declared pass/fail rule, is level.

Why 99% is not a pass

A task counts as passed only when every hidden check passes. Partial credit is the fraction of hidden checks that passed, and it is a diagnostic, not a pass. Two of this battery's eight tasks hinge on that distinction.

hx_minidb_mvcc has 75 hidden checks. Both models pass 74 and fail exactly one: check_32: value_validation. Both build an engine that satisfies 74 of those checks, and both fail the same one, in every trial: 0.987 partial credit, zero passes, from either model. A single shared miss on one check out of 75 looks less like a capability gap and more like an under-specified requirement; it is the first task we would audit before using it to separate models.

hx_api_migration is the same story with a wider gap: 47 or 48 of 66 checks for Terra, 48 for Sonnet, both scoring zero. The failures cluster in the same families — the consent gate's status and body, the invoice-download path's logging — which again points at a specification the models read the same way and the checkers read differently.

Anyone reading only the partial-credit column would conclude that both models nearly solve hx_minidb_mvcc and neither has a real weakness there. They do not solve it, and the honest description of that task is "unsolved by both, one check away".

Where they part company

Rostering: the difference was effort, not insight

hx_rostering is the single largest divergence in the battery, and it is a correctness divergence. Terra scored 14 of 42 hidden checks in all three trials: its schedules break minimum staffing (rule R7) and put staff on units without the required skill (rule R8), cost 1.31 to 2.39 times the reference roster, and carry 438, 2,974 and 3,786 individual rule violations across its 14 instances. It stopped after 10 or 11 turns and 88 to 162 seconds.

Sonnet scored 42 of 42 once and 36 of 42 twice, with labour cost between 0.24 and 0.67 times the reference and zero hard-rule violations in every instance it produced — its two failing trials each crash two of their fourteen instances before scoring, and a crash is not a solved instance. It took 2,504 to 3,069 seconds and 78 to 152 turns per attempt.

The interesting number is not the score, it is the 20-fold difference in work: on the one task where one model produced a legal roster and the other did not, the winning model kept iterating against the constraints for nearly an hour. Terra's output looks like a roster and is not one. This is the failure mode to watch for in agentic engineering work — a plausible artifact delivered early.

It also shows how coarse partial credit is. Terra's three attempts scored exactly 14 of 42 every time, while the number of rule violations inside those attempts varied by a factor of nine — from 438 to 3,786. Identical scores, wildly different artifacts. A partial-credit column cannot see that difference, which is why this article quotes the underlying violations rather than the fraction.

Vulnerability audit: the same audit, failed from different angles

Both models score zero on hx_vuln_audit and their failures sit in different places.

Terra passes 49, 50 and 50 of 56 checks. Every trial it leaves the pickle wire hazard unfixed (X.x_pickle_wire), regresses the write-scope policy (R.r_policy), misses the encode_job deserialisation finding (F.recall.05) and one or both of two auth findings (F.recall.07, F.recall.08), and reports 4 to 6 false positives. In two of three trials it also blocks a legitimate hit outright (R.legit.3 returns nothing where one match was expected). One trial additionally misses a ReDoS.

Sonnet passes 45, 43 and 45 of 56. Every trial it leaves four or five exploit checks failing: always symlink TOCTOU (X.x_toctou), shared-write (X.x_shared_write) and pickle, plus parameter pollution and a missing signature check in two trials each, and an integer overflow in one. It misses six finding classes across the run (F.recall.03, 05, 06, 07, 08, 09). Its false-positive count is lower: 1 to 3. It also blocks a legitimate hit in two of three trials.

So both leave the target exploitable, and both over-block something legitimate. The difference is the shape: Sonnet's misses are live exploit classes that turn into CVE-grade holes — TOCTOU, shared writes, parameter pollution, a missing signature — while Terra's are a stable list of missed findings plus a policy regression, with more false positives and more over-blocking. Neither profile is the safe one for a security review, and the partial score ranks them in the opposite order to the exploit count.

The same wall, different scratches

hx_lang_interp fails the same three generated-expression checks for both models in the trials that fail — a bigint literal that loses precision — at 56 to 57 of 60 checks each. Terra passes 1 of 3 trials there and Sonnet 2 of 3; neither has solved the problem, and which trial flips is a stability question rather than a capability one.

hx_perf_engine is the only task both models solved on every trial: 3 of 3, 42 of 42 checks, 100% partial credit.

Consistency

Repeat behaviour separates them more than raw score does. Terra solves 50% of tasks at least once and 37.5% on every trial; Sonnet solves 62.5% at least once and 12.5% on every trial. Mean per-task trial variance is 0.028 for Terra and 0.111 for Sonnet. Sonnet has the higher ceiling and the wider spread; Terra does the same thing twice as often. For work that runs unattended, that is the number that matters — a model that passes a task two times in three will still fail one night in three.

Time and effort

Sonnet was slower on all eight tasks, by anywhere from 2.3 times on hx_minidb_mvcc to 24.4 times on hx_rostering — 3.5 on hx_lang_interp, 7.2 on hx_fleet_incident, 7.7 on hx_perf_engine. A median trial took 604 seconds against Terra's 129, with an average of about 42 turns per trial against 18, 1.68 million output tokens against 163 thousand, and 155 million cached input-token reads against 18.6 million. On hx_perf_engine both models solved the task perfectly, and Sonnet took 7.7 times as long to do it.

Token accounting between the two providers is not symmetric — Anthropic reports cache reads separately and the OpenAI Codex route does not report them the same way — so the token figures here are indicative of effort, not a like-for-like efficiency measure. Wall clock and turn counts are directly comparable.

Access and cost

Terra's subscription route has no per-token meter to report, so this article publishes no dollar figure for it: quoting one would mean inventing a price for access that is not billed per token. Sonnet ran on the metered Anthropic API, which is billed per token, but no price list is shipped with these runs, so no dollar estimate is published for it either. The cost column reads n/a for exactly that reason — it is a deliberate omission, not a gap in the data.

What this does not show

• One complete run per model, 24 cells each. Repeat runs for both models exist and are incomplete — Sonnet's second run had verified 23 of 24 cells and Terra's first 22 of 24 when this was written — so they are treated as supplementary and excluded from every primary number here. Two complete repeats, which is what a variance claim requires, do not exist yet.
• The interval is wide. A 24-cell Wilson interval runs from 24.5% to 61.2%, which means this data is compatible with a substantial difference in either direction. It cannot support "model A is better than model B".
• One battery. These are hard agentic engineering tasks with hidden checks. Nothing here transfers automatically to writing, long-context recall, or cost-constrained workloads, none of which were run.
• The harnesses are close but not identical. Each run used its own sandboxed copy of the harness, at commits 8746c93 (Terra) and 0d42a0b (Sonnet), and both copies had uncommitted local changes — eleven files in one tree, nine in the other, in each case including the run driver run_eval.py. The task content hashes, the hidden checks and the scoring-rule fields recorded in the two reports are identical, which is what makes the two columns comparable; the driver code is not byte-identical between the sandboxes, and the adapters differ by necessity because they talk to two different providers.
• Partial credit is not quality. It weights every hidden check equally, so a model that satisfies many trivial checks and misses the one that matters looks nearly solved. hx_minidb_mvcc and hx_api_migration are the cases in point.

Reproducing it

The per-trial rows, the merged report and the two immutable run snapshots ship with the article. Every number above comes from the rows: pass counts and per-task partial scores are recomputed from the 48 canonical rows, the pass intervals are Wilson intervals over trials, and the provider attributions are read back from each run's own session store.

Per-track scoreboard

Pass rate over valid graded trials; brackets are 95% Wilson intervals, then passes/n. Computed from rows.jsonl; invalidated (fallback) and errored trials are excluded.

Task heatmap

What the run cost

No per-token price exists for these runs, so no dollar figure is published: one model ran on a flat-rate subscription route with no per-token meter, and the other's metered API pricing is not shipped with the run. Quoting either as a per-token cost would be invented. The tokens actually consumed are in the token table below.

Token efficiency

Medians per scored trial (one trial = one task session, all its API calls). Reasoning is as reported by the provider; 0 can mean hidden rather than absent. Providers report input and cache-read tokens differently (Anthropic, for one, bills cache reads separately, so its median input is the uncached remainder) — compare medians across providers with that in mind.

Latency

p50 / p90 wall-clock seconds per scored trial through the Hermes CLI (nearest-rank: the value at rank ceil(q×n), no interpolation).

How models failed

Every non-passing or excluded trial, bucketed by heuristics on the grader's detail, err and status: fallback-invalid = served by another model (valid=false, any outcome); timeout/error = infrastructure error or the trial carried a transport/timeout error; empty answer = no deliverable (empty reply, iteration cap, a status note instead of the answer); format = output-shape rule broken (word limits, bare-number, JSON/YAML, syntax); wrong logic = everything else (tests or exact match failed).

Methodology & reproducibility

• Harness commit: 8746c93 (terra) / 0d42a0b (sonnet).
• Run: hard1s-terra-vs-sonnet-20261028; 48 task rows and 0 judge rows in rows.jsonl (sha256 08574a2bbe48c97d).
• Tracks: hard: 8 task(s) × 3 trial(s).
• n per model × task cell: 3–3 valid trials (cells with fewer than the scheduled trials lost them to errors or invalidation).
• Scored trial: status pass/fail and not marked invalid. Pass rates use 95% Wilson intervals; percentiles use the nearest-rank convention (value at rank ceil(q×n)).
• Writing judges: none. A win requires the judge to pick the same response in both A/B orders; a split verdict is a tie. Responses are blind (labelled Response 1/2), each judge sees both A/B orders.

Reproduce this page: grading/hb_eval_v3_rows.py <hb-eval tree> <run> (the rows report v3 ranks), then grading/hb_eval_article.py --report report.json --dossier dossier.md --analysis analysis.md --prices prices.json --rows rows.v3.jsonl --harness-commit 8746c93 (terra) / 0d42a0b (sonnet) --out web/articles/<slug>.html.

Method

Run hard1s-terra-vs-sonnet-20261028, report generated 2026-10-28T23:29:44Z. Models compared: claude-sonnet-5-5@anthropic#high and gpt-6.1-terra@openai-codex#high. Tracks scored: Hard battery (8 tasks, 3 trials each).

• Agentic tasks run multi-turn Hermes tool use in a real workspace; hidden tests run after the model finishes. Partial is the mean fraction of hidden tests passed.
• Invalidation. A trial where any call was served by a model other than the one requested is excluded (fallback); an agentic trial whose messages or tool calls reached for the answer key or another agent is excluded (escape). Errored trials are excluded and counted.
• Cost. No per-token price was supplied for these models, so no cost estimate is published: a subscription route has no per-token meter and no API price list ships with these runs. The tokens each model actually consumed are reported in the token tables instead.
• Latency is wall-clock seconds per task through the Hermes CLI, including CLI overhead.

Scored under report v3: only trials whose serving model is verified count; one canonical trial per model, task and trial number (retries and duplicates are counted, never ranked); a track with under 90% of its planned trials eligible is marked ⚑ and kept in the table. eligible/planned < 90% per track is flagged insufficient_coverage (flagged, not dropped; objective_excluding_insufficient shows the effect)