A field guide to frontier model evaluation
What the big labs measure, and how the numbers are actually made
Every model launch from Anthropic, OpenAI, Google, Mistral, and the Chinese labs (Qwen, DeepSeek, Zhipu/Z.ai) ships with a wall of benchmark scores. This atlas explains what each major benchmark really tests, how it is constructed and graded, where it can mislead, and where the current frontier actually stands.
- Benchmarks are an arms race against saturation.Today's "hard" test is next year's solved one. SWE-bench Verified, Aider Polyglot, MMLU-Pro, Humanity's Last Exam and ARC-AGI-2 were each engineered specifically to re-open a score range that frontier models had already maxed out.
- The number is meaningless without the harness.The same model can swing 20+ points depending on the scaffold, tool set, token budget and number of attempts. On ARC-AGI-2, a refinement wrapper lifted Gemini 3 Pro from ~31% to 54%, at roughly 37× the cost per task.
- Lab-reported ≠ independently verified.Labs report their own runs on their own harness, choosing tests they win. Neutral evaluators (Epoch AI, Artificial Analysis, Vals AI, LMArena) frequently land lower. Always ask who ran the eval.
- Contamination is real, and admitted.Public benchmarks leak into training data. ARC Prize itself documented models applying ARC-specific conventions unprompted, evidence the data is baked into the weights.
How to read a benchmark score
Before any specific test, five distortions apply to nearly all of them. Internalising these turns a marketing chart back into information.
- Saturation & the moving goalposts. A benchmark is useful only while it discriminates. Once the field clusters near the ceiling (MMLU at ~90%, MATH-500 at ~95%, AIME 2024 near 100%), differences become noise and the benchmark is quietly retired or replaced by a harder successor.
- Harness / scaffold sensitivity. "GPT-X scores Y%" hides a stack of choices: agent loop, allowed tools, retries (pass@1 vs pass@k / avg@n), thinking budget, and prompt format. A strict output format (GPQA needs an exact Answer: X) can zero out correct reasoning; a richer scaffold can add 20 points.
- Who ran it. Lab-run numbers are an upper bound under favourable conditions. Third-party re-runs on a fixed harness (Epoch AI, Artificial Analysis, Vals AI) are more comparable across models but usually lower.
- Contamination. Public test items end up in pre-training. Mitigations: held-out private sets (FrontierMath, ARC semi-private eval), date-gated problems (LiveCodeBench), and fresh annual sets (AIME 2025 over 2024).
- Self-reported construction claims. Many "robustness" and "human-baseline" figures come from the benchmark's own authors and aren't independently audited. Treat them as design intent, not ground truth.
Can it actually ship software?
These benchmarks moved the field from "write a function" to "operate in a real repo or terminal, autonomously, and be graded by tests." They are the headline numbers Anthropic, OpenAI and the Chinese coding models now lead with.
SWE-bench & SWE-bench Verified
the real-world coding standard Verified- Measures
- End-to-end agentic software engineering: given a real GitHub issue and the whole repository, navigate the code and produce a patch that fixes it without breaking anything else.
- Construction
- Original SWE-bench = 2,294 issue→pull-request tasks mined from 12 popular Python repos (Django, SymPy, scikit-learn…). SWE-bench Verified is a 500-sample human-validated subset: 93 experienced Python developers screened 1,699 random samples, discarding items with under-specified issues, overly-specific tests, or broken environments. It supersedes the original and the "Lite" set.
- Scoring
- % of issues resolved (pass@1). The model's patch is applied in a container and graded by the repo's own hidden unit tests: FAIL_TO_PASS proves the fix works, PASS_TO_PASS proves nothing regressed. Models run inside an agentic harness (bash + file-editor + apply-patch tools, with a token budget) and are told not to edit the tests.
- Caveats
- The original under-stated ability: GPT-4o jumped from 16% → 33% just by cleaning the data. Scores depend heavily on the scaffold and token budget, so cross-lab comparisons need the same harness. Public GitHub means contamination risk; peer-reviewed audits (UTBoost, ICSE 2026) find some "solved" issues are patched incorrectly but still pass weak tests. Harder successors (SWE-bench Pro, Multilingual, Multimodal) now exist because Verified is climbing into the high 80s–90s.
Terminal-Bench
agent in a shell Lab-reported- Measures
- Whether an agent can complete multi-step real tasks in a terminal sandbox (build/compile, debug, configure systems, manipulate files and data), not just emit one patch.
- Construction
- Curated tasks each running in a containerised environment with a programmatic success check. The model drives an interactive shell in a loop. A Terminal-Bench Hard subset isolates the items that still challenge frontier agents. Version 2.x is the current generation labs cite.
- Scoring
- % of tasks completed to the verified end-state. Like SWE-bench it is run through an agent framework, so the wrapping harness matters as much as the model.
- Caveats
- Highly harness-sensitive; the same model posts very different numbers under different agent frameworks. A favourite launch stat for OpenAI (GPT-5.x) and the agentic coding models because long-horizon terminal work showcases tool-use stamina.
LiveCodeBench
contamination-resistant competitive coding Established- Measures
- Algorithmic / competitive-programming ability: code generation, self-repair, test-output prediction and code execution.
- Construction
- Continuously scrapes new problems from LeetCode, AtCoder and Codeforces with known publication dates. You evaluate a model only on problems released after its training cutoff, a clean, contamination-free window that the others lack.
- Scoring
- pass@1 against hidden test cases; reported per time-window so results stay comparable as the problem set grows.
- Caveats
- Scores depend on which date window you quote: pick a window before a model's cutoff and contamination creeps back in. Tests competitive puzzles, which correlate only loosely with everyday engineering. A staple of DeepSeek and Qwen launches.
Aider Polyglot
edit-format discipline across 6 languages Verified- Measures
- Whether a model can produce a correct code edit and emit it in a precise diff format, so it tests both coding and strict instruction-following.
- Construction
- 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust, selected from 697 as those solved by ≤3 of a ~7-model reference panel, deliberately calibrating difficulty.
- Scoring
- % solved (up to two attempts) and % of responses in a well-formed edit format; graded by the exercises' unit tests. The scale was reset so top models land in a wide ~5–50% range. At launch (Dec 2024) OpenAI's o1 topped it at 61.7% in diff format.
- Caveats
- Built precisely because Aider's previous benchmark saturated past 80%. A malformed diff fails even with correct logic, which rewards format compliance. Static set → contamination accrues over time.
Agentic tool-use
τ-bench · BFCL · agent suites Established- Measures
- Multi-turn function/tool calling and policy adherence in simulated agent settings (e.g. a customer-service or retail workflow): does the model call the right APIs, in order, and respect rules?
- Construction
- Sandboxed environments with a tool set and a simulated user. τ-bench (tau-bench) and the Berkeley Function-Calling Leaderboard (BFCL) are the common references. Success = correct final world-state plus policy compliance, often over multiple turns.
- Scoring
- Task success rate, sometimes a "pass^k" stability metric (does it succeed on all k independent tries, a reliability stress test).
- Caveats
- Simulated users imperfectly mimic real ones; results swing with prompt and tool-schema design. Anthropic leans on these to argue Claude's agent reliability.
Can it think, not just recall?
The prestige tier. These are where OpenAI, Google and DeepSeek stage their "frontier reasoning" claims: PhD science, olympiad math, and pure abstraction designed to resist memorisation.
GPQA Diamond
Google-proof PhD science Verified- Measures
- Graduate-level scientific reasoning in biology, chemistry and physics, questions hard enough that web search doesn't help.
- Construction
- The 198-question "Diamond" subset (chem 93 / phys 86 / bio 19): the hardest tier, where both expert validators agreed on the answer but most non-experts got it wrong. Skilled non-experts with unrestricted internet score only ~34%; domain PhDs ~65–74%.
- Scoring
- pass@1 accuracy, 25% random-guess baseline. OpenAI's simple-evals prompt demands a strict Answer: $LETTER (A–D); a regex extracts it and mis-formatted answers score zero.
- Caveats
- Only 198 items, so a handful of questions swing the percentage: treat ±2 points as noise. Four-choice multiple choice is partly guessable and shortcut-able. Now saturating (frontier models ~90%+), which is why labs increasingly pair it with HLE and FrontierMath.
AIME 2024 / 2025
olympiad-track high-school math Established- Measures
- Hard multi-step competition mathematics (American Invitational Mathematics Examination), the qualifier above the AMC, below the USAMO.
- Construction
- Just 30 problems per year (two 15-question papers); every answer is an integer 0–999. Labs use the freshest year (AIME 2025) to dodge the contamination that has soaked AIME 2024.
- Scoring
- % correct, almost always averaged over many samples (e.g. avg@32 / cons@64) because per-run variance is huge with only 30 items.
- Caveats
- Tiny set → very high variance; one problem is ~3.3 points. Integer-only answers mean a lucky guess or a shortcut can land without sound reasoning. AIME 2024 is essentially saturated (top models near 100%); AIME 2025 is the live target.
MATH-500
the now-easy baseline Established- Measures
- Competition math across algebra, geometry, number theory and more, at a range of difficulty levels.
- Construction
- A 500-problem slice of the Hendrycks MATH dataset, curated by OpenAI for the "Let's Verify Step by Step" work. Final-answer accuracy, auto-graded.
- Scoring
- % of final answers correct (pass@1).
- Caveats
- Largely saturated (frontier ~95%+) and old/public, so contamination is a given. Survives as a cheap sanity-check baseline, not a frontier discriminator, which is exactly why the field moved to AIME 2025 and FrontierMath.
ARC-AGI & ARC-AGI-2
fluid intelligence, no knowledge Verified- Measures
- Fluid intelligence / abstraction: infer a transformation rule from a few example grid pairs and apply it to a new input. By design it needs no world knowledge or language, only "core knowledge" priors.
- Construction
- Coloured input→output grids (1×1 to 30×30, 10 colours); every task is unique and unmemorisable. ARC-AGI-2 (2025) is the much harder successor. It is confirmed human-solvable (100% of tasks were solved by ≥2 non-expert testers), yet brutal for AI. Evaluated on a semi-private set to limit overfitting, with cost-per-task reported alongside accuracy.
- Scoring
- % of held-out tasks solved exactly. The difficulty cliff is the story: o3 (Medium) scored 53% on ARC-AGI-1 but only 3.0% on ARC-AGI-2 (May 2025). By late 2025 base models reached the 30s: Claude Opus 4.5 at 37.6% (~$2.20/task), Gemini 3 Pro ~31% (~$0.81/task).
- Caveats
- Scaffolds inflate scores at extreme cost: a Poetiq refinement wrapper pushed Gemini 3 Pro to 54% at ~$30/task, capability of the system, not the base model. ARC Prize even reports contamination signs: a model emitted correct ARC colour conventions the harness never mentioned. Beware third-party leaderboards quoting 70–98% on ARC-AGI-2: those contradict the verified ARC Prize numbers and are not credible.
Humanity's Last Exam (HLE)
the "final" academic exam Verified- Measures
- The frontier of closed-ended expert knowledge and reasoning across 100+ subjects, explicitly built to be the hardest academic exam models can't yet ace.
- Construction
- 2,500 questions from nearly 1,000 subject-matter experts across 500+ institutions in 50 countries; a mix of multiple-choice and exact-match short answer, including a multimodal slice. Created directly to counter the saturation of benchmarks like MMLU.
- Scoring
- pass@1 exact-match / MCQ accuracy. Crucially, results split into "with tools" (web/search) vs "text-only": these differ a lot, so only compare like-for-like.
- Caveats
- Launched with frontier models in the single digits; climbing fast (reported into the 40s–50s by mid-2026, higher with tools). Known critiques about answer-leakage on the multiple-choice portion. The "with tools" framing is a frequent apples-to-oranges trap in launch charts.
FrontierMath
research-grade mathematics Verified- Measures
- Mathematics far beyond AIME/MATH: from advanced undergraduate up to genuine research-level problems.
- Construction
- ~338 unpublished, original problems (Epoch AI), authored by professional mathematicians. Tiers 1–3 span undergrad→graduate; Tier 4 is research-level. Only ~12 are public: the rest are held out to defeat contamination.
- Scoring
- Exact-answer accuracy with automated (often SymPy) verification: answers are constructed to be checkable but hard to guess.
- Caveats
- Carries a transparency controversy: Epoch's OpenAI funding was initially undisclosed, an o3 result showed a discrepancy (~10% independent vs ~25% claimed), and ~42% of v1 problems reportedly contained errors. Held-out design means you must trust the evaluator. Scores stay low even for frontier models, which is the whole point.
Breadth, languages, and following instructions
The "general capability" backbone. Several are saturated and survive mostly as comparison baselines, which is itself a useful signal about where the field has moved.
MMLU & MMLU-Pro
the broad-knowledge yardstick, then its hardened sequel Verified- Measures
- Broad multitask knowledge and reasoning across academic and professional domains.
- Construction
- MMLU (2020): ~14,000 four-choice questions over 57 subjects. MMLU-Pro: 12,000+ questions consolidated into 14 domains, with the answer set expanded from 4 to 10 choices (3× more distractors) and more reasoning-heavy items, dropping the guess baseline from 25% → 10%.
- Scoring
- pass@1 accuracy. MMLU-Pro is both harder (a 16–33% absolute accuracy drop vs MMLU) and more robust: prompt-format sensitivity falls from 4–5% down to ~2%, so rankings are more trustworthy.
- Caveats
- Plain MMLU is saturated (frontier ~88–92%) and carries known label errors (~6.5% in audits like MMLU-Redux), so small gaps at the top are meaningless. Both remain four-/ten-choice multiple choice and are vulnerable to shortcut exploitation. MMLU-Pro's robustness figures are author-reported.
Multilingual (MMMLU) & long-context (RULER, NIAH)
across languages and across 1M tokens Established- Measures
- MMMLU: knowledge transfer across languages (OpenAI's MMMLU is MMLU professionally translated into 14 languages). Long-context evals: whether a model actually uses a 100k–1M-token window rather than just accepting it.
- Construction
- Multilingual: per-language accuracy, then averaged. Long-context: Needle-in-a-Haystack plants a fact at varying depths/lengths and asks for it back; RULER adds synthetic retrieval, multi-hop and variable-tracking tasks that scale with context length.
- Scoring
- Accuracy vs language, or accuracy vs context length. RULER's key output is the "effective context", the length beyond which a model degrades.
- Caveats
- Needle-in-a-Haystack is largely saturated (an easy retrieval task). RULER routinely shows effective context far below the advertised window. Translated benchmarks carry translation artifacts and still favour English. Google and Qwen lean on long-context claims (1M+ windows) heavily at launch.
IFEval
does it follow the literal instruction Established- Measures
- Verifiable instruction-following: "write ≤3 paragraphs", "include the word X twice", "reply in JSON", "use no commas". Format compliance, not answer quality.
- Construction
- ~500 prompts carrying 25 types of programmatically checkable instructions, so grading is objective code, no human or LLM judge in the loop.
- Scoring
- Strict and loose accuracy, reported at both prompt level (all instructions met) and instruction level (per-constraint).
- Caveats
- Only tests obedience to format, not correctness or usefulness: a model can ace IFEval and still be wrong. Saturating at the top (~85–90%+). A near-universal entry on instruction-tuned model cards because it's cheap and objective.
Seeing as well as reading
A brief tour of the headline vision-language benchmark. Google in particular foregrounds these, since native multimodality is a core part of Gemini's pitch.
MMMU & MMMU-Pro
college-level multimodal reasoning Verified- Measures
- Expert-level reasoning over images + text (diagrams, charts, tables, chemical structures, medical scans, music notation) across 30 subjects.
- Construction
- ~11.5k questions spanning 183 subfields and 30 image types, drawn from college exams and textbooks; multiple-choice plus open-ended. MMMU-Pro is the harder, vision-robust successor that closes a known loophole.
- Scoring
- Accuracy. MMMU-Pro adds a vision-only setting and more answer options to force genuine image use.
- Caveats
- On plain MMMU, some questions are answerable from the text alone, a vision shortcut MMMU-Pro was built to remove. Saturating for top multimodal models, so MMMU-Pro is the figure to watch.
Where the frontier stands: mid-June 2026
A point-in-time picture, not a leaderboard. Methodology above is stable; these numbers are volatile and come from mixed sources. Read every figure together with its badge, and never compare across columns sourced from different harnesses.
⚠ Read this before the tables
Cross-source benchmark comparison is genuinely unreliable. Labs report their own best runs; aggregators re-run on different harnesses. Treat Lab-reported as an optimistic upper bound and Third-party as indicative only.
Two concrete warnings. (1) GLM 5.2 shipped with no official benchmarks at all: every GLM 5.2 number below is third-party and should be treated as provisional. (2) Several aggregators list ARC-AGI-2 scores of 70–98% for current models; these flatly contradict the verified ARC Prize data (best base model ≈ 37% in late 2025) and are Disputed. We exclude them.
The current flagships
| Lab | Flagship (mid-Jun 2026) | Released | Note |
|---|---|---|---|
| OpenAI | GPT-5.5 | Apr 2026 | Leads with agentic/terminal & frontier-reasoning framing |
| Anthropic | Claude Opus 4.8 | May 2026 | Primary served flagship; Claude Fable 5 (Jun) is newer but reportedly restricted |
| Google DeepMind | Gemini 3.1 Pro / 3.5 Flash | Feb / May 2026 | Broad model-card sweep + multimodal + long-context |
| Mistral | Mistral Medium 3.5 | Apr 2026 | Efficiency / open-weight / multilingual positioning |
| Alibaba Qwen | Qwen 3.7 Max | May 2026 | Open-weight, 1M context, broad benchmark sweep |
| DeepSeek | DeepSeek V4 Pro | Apr 2026 | Reasoning + cost-efficiency; no R2 yet |
| Zhipu / Z.ai | GLM 5.2 | Jun 13 2026 | 744B MoE (~40B active), 1M ctx, MIT weights, no official benchmarks at launch |
Headline scores (as reported)
| Model | SWE-bench Verified | GPQA Diamond | MMLU-Pro | HLE | Source |
|---|---|---|---|---|---|
| Claude Opus 4.8 | 88.6L | 93.6L | – | 57.9L,~ | Lab-reported |
| Claude Fable 5 | 95.03 | 92.63 | – | 53.33 | Third-party |
| GPT-5.5 | – | – | – | – | Third-party |
| Gemini 3.1 Pro | 80.6L | 94.3L | 92.6L | 44.7L | Lab-reported |
| Qwen 3.7 Max | 80.4L | 92.4L | – | – | Lab-reported |
| DeepSeek V4 Pro | 80.63 | – | – | – | Third-party |
| Mistral Medium 3.5 | 77.6L | – | 80.7L | – | Lab-reported |
| GLM 5.2 | 62.13,§ | 91.23 | – | 54.73,~ | Third-party |
§ GLM 5.2's coding figure is SWE-bench Pro (a harder, held-out variant), not directly comparable to the SWE-bench Verified column above. HLE figures marked ~ mix "with-tools" and "text-only" settings across sources, a known apples-to-oranges hazard. GPT-5.5's launch numbers circulating in mid-June were Terminal-Bench (~82.7) and SWE-bench Pro (~58.6) rather than SWE-bench Verified, so its Verified cell is left blank rather than guessed.
ARC-AGI-2: the one current-score set that survived verification
This is the clearest illustration of why cost and harness belong next to every score. The verified ARC Prize data (late 2025 / early 2026) tells a very different story from the inflated aggregator numbers.
| System | ARC-AGI-2 | ≈ $/task | Type |
|---|---|---|---|
| Human (non-expert panel) | 100% solvable | – | baseline (≥2 testers solved every task) |
| Poetiq + Gemini 3 Pro | 54% | ~$30 | refinement scaffold |
| Claude Opus 4.5 (Thinking 64k) | 37.6% | ~$2.20 | base model |
| Gemini 3 Pro | ~31% | ~$0.81 | base model |
| o3 (Medium), May 2025 | 3.0% | – | base model (53% on ARC-AGI-1) |
The takeaway in one line
A scaffold roughly doubled the score (31% → 54%) for about 37× the cost ($0.81 → $30 per task). When a launch chart shows a big ARC-AGI-2 number, the first question is always: base model, or expensive wrapper?
Which benchmarks each lab leads with, and why
A launch benchmark slate is a marketing artifact: labs foreground the tests they win and that match their product story. The selection itself is the signal.
Anthropic agentic coding
Leads with SWE-bench Verified, terminal/agent tasks and tool-use reliability (τ-bench). GPQA and HLE feature, but coding is the headline.
Reveals: a deliberate "model that does real engineering work" enterprise position: the Claude Code / agent story.
OpenAI frontier reasoning
Foregrounds FrontierMath, ARC-AGI, competition math (AIME/IMO) and PhD-level GPQA, alongside SWE-bench and Terminal-Bench for agents.
Reveals: an AGI / raw-reasoning framing: "smartest model" more than "best worker."
Google DeepMind generalist + multimodal
Publishes the broadest model-card spread: MMLU-Pro, GPQA, AIME, SWE-bench, MMMU, long-context to 1M+, plus LMArena Elo.
Reveals: a "most capable all-rounder, natively multimodal, huge context" pitch; heavy emphasis on human-preference ranking.
Mistral efficient & open
Emphasises MMLU/MMLU-Pro, coding and multilingual (European languages) framed around cost-performance and open weights.
Reveals: an efficiency / sovereignty / open-weight value position rather than a raw-frontier claim.
Alibaba Qwen open-weight parity
Runs broad leaderboard sweeps (GPQA, AIME, LiveCodeBench, SWE-bench, multilingual, long-context, agentic) to show open weights matching closed frontier.
Reveals: "frontier capability you can download," aimed squarely at the closed labs.
DeepSeek reasoning at low cost
Centres AIME, MATH, GPQA, Codeforces/LiveCodeBench, SWE-bench plus the economics of training and inference.
Reveals: "frontier reasoning at a fraction of the cost": the efficiency-disruptor narrative.
Zhipu / Z.ai (GLM) open agentic coding
GLM 5.2 highlighted long-horizon coding (Terminal-Bench, SWE-bench Pro), design tasks, a usable 1M context and price (~⅙ of GPT-5.5), and shipped with no official benchmarks at all, leaning on availability, MIT weights and third-party evals.
Reveals: an open-weight, agentic-coding value play; the absence of a lab benchmark slate is itself a positioning choice.
The common pattern read critically
Everyone shows the tests they win, on their own harness, in their own framing. The fix is the same every time:
- Check a neutral third party (Epoch AI, Artificial Analysis, Vals AI, LMArena).
- Ask whether the eval was lab-run and with what scaffold.
- Prefer contamination-resistant tests (held-out, date-gated, fresh-year).