GPT-6.1 Terra went 90 for 90 on our fleet benchmark. So did Sol 6.1 and Opus 5.5, at about four times the price

Scoreboard

Highlighted row: subject model. Writing W-L-T is the row model's wins, losses and ties against the subject, one decision per judge per task (A win requires the judge to pick the same response in both A/B orders; a split verdict is a tie.). Objective = b1 + code + ops + context tracks pooled.

⚑ Report v3 flags. MiniMax-M3.1-Flash-Preview: low coverage: agentic tool use 7/10; 3 unverified-attribution trials not counted. grok-4.7: low coverage: agentic tool use 6/10, hard coding 4/24; 1 unverified-attribution trial not counted.

Results by track

Cost versus score

Every task, every model

Cells show passes / valid trials; colour runs red (0%) to green (100%).

Update, October 28

We added the two flagship models every benchmark here now includes, GPT-6.1 Sol and Claude Opus 5.5, and re-scored all nine models under a stricter rule set (report v3, below). The objective scores of the seven models in the first edition did not change. Three things did: the headline is no longer Terra alone; Terra no longer wins the writing comparison against every rival; and two scores now carry a ⚑ low-coverage flag, because fewer than 90% of their planned trials could be counted.

Three corrections to how those numbers were produced, so you can check them:

• 18 of Sol 6.1's trials were first graded on a rate-limit notice. Our provider answered some calls with a "too many requests" message as if it were the model's reply, and the grader marked those as failures. We changed the harness to treat such a reply as a transport error, re-ran those 18 trials, and Sol 6.1 passed all of them.
• Opus 5.5's agentic trials were wrongly flagged as cheating. Our integrity check looked for any path inside the agent profiles folder in a transcript. Our own harness points each model's scratch directory there, so a model that wrote a helper script to its scratch space tripped the check. That was 15 of Opus 5.5's 20 agentic transcripts; none touched the hidden tests or another agent. We fixed the check (a model's own scratch space no longer counts; the answer key, other agents and .. escapes still do), added a test, and re-checked every agentic transcript of all nine models: the 15 Opus flags were the only change.
• One judge changed. DeepSeek V4.1 Flash judged part of the first edition's writing. For the Sol 6.1 and Opus 5.5 comparisons its key was refused, so MiniMax M3.1 Flash took its place. Under report v3, 3 of 129 judge votes were not counted (two invalid, one whose serving model could not be confirmed), leaving 126.

We also re-ran Terra from scratch in a sealed copy of the harness: it again passed 90 of 90, with no task changing its result.

What we tested

Hermes Benchmarks runs an autonomous fleet of AI agents. The New Model Evaluation Protocol tests what those agents do every day, not trivia. It has six tracks:

• Daily suite: 11 short coding and reasoning tasks, the same ones that grade our fleet.
• Hard coding: 8 harder one-shot Python modules (a cron schedule calculator, a semver range matcher, a unified-diff applier, a tiny JSONPath engine and others), each checked by 10 to 31 hidden assertions.
• Ops and format compliance: 8 tasks where the answer must come back in an exact machine-readable shape: a bare number, strict YAML, bullets under a word limit.
• Long-context recall: a single fact hidden in roughly 60,000, 200,000 and 500,000 characters of text.
• Agentic tool use: 5 small real workspaces with bugs to fix, a feature to add, logs to triage, a config to migrate and code to refactor. The model works with real terminal and file tools, then hidden tests it never saw decide the grade.
• Writing: 4 briefs (an operator status update, a model release brief and others) judged blind by two models from labs other than the two contestants, in both A/B orders. A win counts only if a judge picks the same answer both ways round; otherwise it is a tie.

Every task ran three times (agentic tasks twice). Under report v3 a trial is ranked only if our own records confirm which model answered it; trials served by a fallback model are thrown out, and trials whose serving model we could not confirm are counted and shown, never ranked. Each model gets one trial per task and trial number; retries are counted but never ranked. A track where fewer than 90% of the planned trials could be counted is marked ⚑ and kept in the table. Any agentic trial whose transcript showed it reaching for the answer key or handing the job to another agent is thrown out.

Where Terra is good

It does exactly what it is told. The hard-coding track is where the field split. Terra passed 24 of 24, as did Sol 6.1, Opus 5.5 and Claude Sonnet 5.5. GPT-6 Sol passed 19, and all five of its failures had the same cause. The task asked for the module as the reply. Sol instead used the file tools it had been given to write the module to disk, then replied with only a status note ("I did not run tests or verify that it imports … so it is not a verified deliverable"). The grader received no code. DeepSeek V4.1 Flash lost two tasks to stray characters inside its code, a leftover code fence and an em dash.

It is efficient. Terra wrote the fewest output tokens of any model on graded tasks: a median of 145 per trial. Sol 6.1 wrote 161, Opus 5.5 250, DeepSeek V4.1 Flash 660 and MiniMax M3.1 Flash 989 (most of it reasoning). Fewer tokens is why Terra's per-token price turns into a low price per finished task.

It is fast at agentic work. Its median agentic task finished in 41 seconds, the quickest in the field, and it fixed every workspace on every trial. Opus 5.5 took a median of 52 seconds and Sol 6.1 77 seconds for the same clean result.

Where Terra is weak, or where we could not tell

We could not find its limits. Three models are at 100%, and long-context recall and ops compliance were passed by nearly everyone. The test set needs harder tasks before it can rank the top of the field, and we will add them.

It is not the best writer. Opus 5.5 is the only model that beat Terra in blind writing judgment, and Sol 6.1 drew with it. Against everyone else Terra won.

Its interval overlaps the rivals'. Terra's 95% interval runs from 96% to 100%. DeepSeek V4.1 Flash's runs from 89% to 98%. With 90 trials each, the difference between them is suggestive, not proven.

Our harness gives every model tools, even on one-shot tasks. Each model runs as a real fleet agent, with a small tool set always present. GPT-6 Sol and MiniMax M3.1 Flash lost most of their coding points by reaching for those tools instead of answering. In our fleet that is a real failure, but a harness without tools could score them higher.

We tested one setting. We ran every model at its default reasoning effort inside our own agent harness. We did not test maximum effort levels, long-prompt pricing above 272,000 tokens, or the computer-use features OpenAI's launch focused on.

The rest of the field

• GPT-6.1 Sol (100%) is OpenAI's flagship, and on this suite it bought nothing over Terra but a writing draw. It cost about four and a half times as much per task and was slower at agentic work.
• Claude Opus 5.5 (100%) had the fastest median response in the field, at 21.2 seconds, and is the only model that out-wrote Terra. It cost about $12.45 per thousand tasks, nearly four times Terra.
• DeepSeek V4.1 Flash (95.6%) is our fleet's default model, and this result keeps it there. It failed four graded tasks: two formatting slips and two real logic errors (a diff applier that rejected a valid "no newline at end of file" marker, and a semver edge case with pre-release tags).
• Claude Sonnet 5.5 (94.4%) matched Terra on hard coding, 24 of 24, and answered almost as fast as Opus, at a median of 22.3 seconds. It lost all its graded points on format rules: three times it showed its arithmetic when asked for a bare number, and twice it wrote bullets over the word limit.
• GPT-6 Sol (94.4%) is the previous flagship. Its habit of writing files and reporting on them, when asked to return the code, is a problem for an agent fleet that expects the answer in the reply.
• GPT-5.6 Terra (94.4%) is the model Terra replaces. It was slower, cost about five times as much per task on this suite, and did not win a single decided writing comparison against its successor.
• MiniMax M3.1 Flash Preview (91.1%) is the cheapest model here at about $2.54 per thousand tasks. It passed all 7 of its agentic trials that count; 3 more passed, but we could not confirm which model served them, so its agentic score carries a ⚑ (7 of 10 planned trials). It was weakest at one-shot coding (16 of 24): four times it spent its only turn on a tool call and ran out of steps without answering, and four times its code was wrong.
• Grok 4.7 (95.7%, low coverage) could not be measured properly. The connection we used to reach it was repeatedly answered by a different model under load, so those trials were thrown out. Only 70 of its 90 graded trials count, only 4 of its 24 hard-coding trials and 6 of its 10 agentic trials, and one more trial could not be attributed. Treat its score as unmeasured, not as a result.

What it means for our fleet

We will use GPT-6.1 Terra as the OpenAI model for agent work and as a strong second opinion on hard coding tasks. DeepSeek V4.1 Flash stays the default: it costs the same per task here, and the gap between them is within the noise of this suite. Opus 5.5 and Sol 6.1 earn their price only where writing quality matters. Every new model from the frontier and Chinese labs will now go through this same protocol, against the same three workhorses, and we will publish each result here.

Per-track scoreboard

Pass rate over valid graded trials; brackets are 95% Wilson intervals, then passes/n. Computed from rows.jsonl; invalidated (fallback) and errored trials are excluded.

Task heatmap

What the run cost

Every task row with usage is billed, including errored and invalidated trials and writing generations (the money was spent); judge calls are not. Cache reads are billed at the cache-read rate; where a provider reports cache larger than input, input is taken as the uncached remainder. Reasoning tokens are assumed inside output.

Token efficiency

Medians per scored trial (one trial = one task session, all its API calls). Reasoning is as reported by the provider; 0 can mean hidden rather than absent.

Latency

p50 / p90 wall-clock seconds per scored trial through the Hermes CLI (lower nearest-rank percentile).

How models failed

Every non-passing or excluded trial, bucketed by heuristics on the grader's detail, err and status: fallback-invalid = served by another model (valid=false, any outcome); timeout/error = infrastructure error or the trial carried a transport/timeout error; empty answer = no deliverable (empty reply, iteration cap, a status note instead of the answer); format = output-shape rule broken (word limits, bare-number, JSON/YAML, syntax); wrong logic = everything else (tests or exact match failed).

Writing: pairwise matrix

Row model's wins-losses-ties against the column model; colour is the row's win share (ties count half). A win requires the judge to pick the same response in both A/B orders; a split verdict is a tie. Pairs never judged against each other show –. Incomplete decisions (an A/B order missing or judge-invalid): 2.

Methodology & reproducibility

• Harness commit: 96c0518.
• Run: gpt-6.1-terra-20261027; 909 task rows and 126 judge rows in rows.jsonl (sha256 cfb15463a483ac09).
• Tracks: b1: 11 task(s) × 3 trial(s); code: 8 task(s) × 3 trial(s); ops: 8 task(s) × 3 trial(s); context: 3 task(s) × 3 trial(s); agentic: 5 task(s) × 2 trial(s); writing_gen: 4 task(s) × 1 trial(s).
• n per model × task cell: 1–3 valid trials (cells with fewer than the scheduled trials lost them to errors or invalidation).
• Scored trial: status pass/fail and not marked invalid. Pass rates use 95% Wilson intervals; percentiles use the lower nearest-rank convention.
• Writing judges: MiniMax-M3.1-Flash-Preview@minimax (8 votes), claude-sonnet-5-5@anthropic (48 votes), deepseek-v4.1-flash-1017@surplus (16 votes), grok-4.7@xai-oauth (54 votes). A win requires the judge to pick the same response in both A/B orders; a split verdict is a tie. Responses are blind (labelled Response 1/2), each judge sees both A/B orders.

Reproduce this page: grading/hb_eval_v3_rows.py <hb-eval tree> <run> (the rows report v3 ranks), then grading/hb_eval_article.py --report report.json --dossier dossier.md --analysis analysis.md --prices prices.json --rows rows.v3.jsonl --harness-commit 96c0518 --out web/articles/<slug>.html.

Method

Run gpt-6.1-terra-20261027, report generated 2026-10-28T06:42:34Z. Subject model: gpt-6.1-terra@openai-codex. Tracks scored: Daily suite (b1), Hard coding, Ops / format compliance, Long-context recall, Agentic tool use.

• Objective tracks (b1, code, ops, context) are graded by hidden unit tests, Python checkers or exact string match; valid trials per task in this report: 2/3. The objective score pools every valid trial; the interval is a 95% Wilson interval.
• Agentic tasks run multi-turn Hermes tool use in a real workspace; hidden tests run after the model finishes. Valid trials per task in this report: 1/2. Partial is the mean fraction of hidden tests passed.
• Writing is judged blind and pairwise against the subject's output by up to two judges from labs other than either contestant's, each in both A/B orders; responses are labelled Response 1/2 and model names never reach a judge. A win requires the judge to pick the same response in both A/B orders; a split verdict is a tie. 126 judge verdict(s) recorded.
• Invalidation. A trial where any call was served by a model other than the one requested is excluded (fallback); an agentic trial whose messages or tool calls reached for the answer key or another agent is excluded (escape). Errored trials are excluded and counted.
• Cost is an estimate: mean tokens per non-agentic task (all API calls in the task's session) × price per 1M tokens (cache reads at the cached rate), × 1,000. Agentic tasks are not included. Shown as n/a where no price was supplied.
• Latency is wall-clock seconds per non-agentic task through the Hermes CLI, including CLI overhead.

Scored under report v3: only trials whose serving model is verified count; one canonical trial per model, task and trial number (retries and duplicates are counted, never ranked); a track with under 90% of its planned trials eligible is marked ⚑ and kept in the table. eligible/planned < 90% per track is flagged insufficient_coverage (flagged, not dropped; objective_excluding_insufficient shows the effect)