LLM benchmark cheat sheet: what each major benchmark actually tests and which ones matter for real-world performance

TL;DR

Most benchmark scores answer a narrow, specific question — not the one you actually care about. HumanEval measures single-function code gen, not engineering skill. Chatbot Arena measures conversational charm, not factual accuracy. This cheat sheet tells you what each of 18 major benchmarks actually tests, what it misses, and whether a strong score predicts real-world performance (most don’t). Use it to spot selective reporting before you trust a model launch chart.

Benchmark scores are the first thing you see in a model launch. A wall of charts, a green arrow, a claim of “state of the art.” The charts are usually real, but they answer narrower questions than most buyers assume.

This cheat sheet covers 18 major benchmarks in a quick-reference format. For each one you get:

• What it measures — the actual task the model is asked to perform.
• What it misses — the capability gaps that the benchmark does not test.
• Real-world correlation — whether a strong score here means the model will perform on your workload.
• Saturation risk — whether top models cluster at the ceiling, making small differences noise.

Use this alongside the deeper guides on how LLM benchmarks work and what they miss, benchmark leaderboards for busy buyers, and our coding benchmarks explained guide.

At a glance: benchmark categories

Knowledge & reasoning

MMLU-Pro (Measuring Massive Multitask Language Understanding — Pro version)

What it measures: Answering multiple-choice questions across 57 academic subjects (law, medicine, physics, history, etc.) at undergraduate level. MMLU-Pro adds more challenging distractors and harder questions than the original MMLU.

What it misses: Open-ended reasoning, multi-step synthesis, and domain specialisation beyond undergraduate breadth. A model that scores 90% on MMLU-Pro can still fail to reason correctly about a novel problem that does not fit neatly into a multiple-choice format.

Real-world correlation: 🟡 Moderate. Strong MMLU-Pro scores correlate with general knowledge breadth but not with task-specific reliability. Use it as a floor check — if a model scores poorly on MMLU-Pro, it probably lacks broad capability. If it scores well, you still need workload-specific testing.

Saturation risk: 🟡 Medium. Original MMLU is near saturation for frontier models. MMLU-Pro extends the ceiling but contamination is a known concern.

GPQA (Graduate-Level Q&A)

What it measures: Domain-expert-written multiple-choice questions in biology, physics, and chemistry, designed to be difficult enough that PhDs in adjacent fields cannot easily answer them without specialisation.

What it misses: Cross-domain reasoning, explanation quality, and the ability to apply knowledge outside an academic question format. GPQA is narrow — three sciences only — and tells you little about legal, medical, or business reasoning.

Real-world correlation: 🟢 Good for domain depth. A model that scores well on GPQA demonstrates genuine specialist-level understanding in the tested sciences. But the correlation only holds in the tested domains.

Saturation risk: 🟢 Low. Designed to be difficult enough to separate top models. Currently the hardest knowledge benchmark.

SimpleQA (Factual Accuracy Benchmark)

What it measures: Short-answer factual questions across encyclopedic knowledge (history, science, geography, pop culture). Strict exact-match scoring penalises both hallucination and hedging.

What it misses: Subtlety, nuance, and cases where “the right answer depends on context.” SimpleQA rewards short, precise, verifiable facts — the opposite of what you want from a conversational AI.

Real-world correlation: 🟡 Moderate. Useful for identifying models that hallucinate confidently, but a high SimpleQA score does not mean the model answers nuanced questions well. Low SimpleQA is a red flag; high SimpleQA is not a green light.

Saturation risk: 🟢 Low. Harder questions continue to separate models.

Coding

HumanEval (OpenAI, 2021)

What it measures: Writing a single Python function from a docstring. 164 problems, pass/fail based on unit tests. Tests the narrowest slice of coding: can the model produce a correct function given a clear specification and a standard API?

What it misses: Multi-file editing, understanding unfamiliar codebases, security awareness, debugging, testing, project-level reasoning. A model can score 90% on HumanEval and write insecure, unmaintainable code in a real repo.

Real-world correlation: 🔴 Weak. HumanEval scores have no demonstrated correlation with developer productivity, code quality in real repositories, or ability to handle multi-file engineering work. This is the most over-interpreted benchmark in the industry.

Saturation risk: 🔴 High. Most frontier models score >85%. Differences above 85% are often noise or prompt-tuning artifacts.

See our full coding benchmarks explained guide.

MBPP (Google, 2021)

What it measures: 974 single-function programming problems, easier than HumanEval. Same format — docstring to function — but with simpler specifications and more common patterns.

What it misses: Same blind spots as HumanEval. Easier problems make MBPP scores even less informative for distinguishing top coding models.

Real-world correlation: 🔴 Weak. Scores tend to be higher than HumanEval for the same model, making it a poorer discriminator. A high MBPP score is table stakes, not a differentiator.

Saturation risk: 🔴 High. Ceiling effects visible even in medium-sized models.

SWE-bench (Princeton, 2023)

What it measures: Editing a real GitHub repository to fix an issue. 2,294 real issues from 12 popular Python repositories. The model’s patch must pass the existing test suite.

What it misses: Whether the fix is the most maintainable approach, whether it introduces new security vulnerabilities, and whether the model can work collaboratively with a human. SWE-bench also measures the scaffolding (agent framework, tools, retries) as much as the model — a model can score 10 points higher just by using better tools.

Real-world correlation: 🟡 Moderate. Closer to real development than HumanEval or MBPP, but still an artificial measure. A model that scores well on SWE-bench with a realistic agent setup is more useful than one that only dominates single-function benchmarks.

Saturation risk: 🟢 Low. Still separating models well. Significant headroom remains.

⚠️ Highly scaffold-dependent. Always check the evaluation setup — the same model can score 30% with direct generation and 50% with agent tooling. Compare setups, not just numbers.

Math & logic

GSM8K (Grade School Math 8K)

What it measures: Grade-school-level math word problems requiring multi-step reasoning. The model must show its working (chain-of-thought) and produce a final numeric answer.

What it misses: University-level math, symbolic reasoning, proof construction, and any math that requires formal notation or domain-specific knowledge. GSM8K tests whether a model can follow a simple reasoning chain, not whether it is good at math.

Real-world correlation: 🟡 Moderate. Useful as a sanity check for basic multi-step reasoning ability. If a model struggles with GSM8K, it will likely struggle with any structured reasoning task. If it passes easily, you still need harder tests.

Saturation risk: 🔴 High. Many models score 95%+. Contamination is widespread — GSM8K problems appear frequently in training data.

MATH (Mathematics Aptitude Test of Heuristics)

What it measures: Competition-level math problems (AMC, AIME, Olympiad style) across algebra, geometry, number theory, probability, and calculus. Requires genuine mathematical reasoning, not pattern matching.

What it misses: Applied math in business contexts, statistical reasoning over real data, and the ability to explain mathematical concepts to non-experts. MATH is a theoretical muscle test, not a practical one.

Real-world correlation: 🟢 Good for structured reasoning. A strong MATH score predicts the model can handle complex, multi-step logical chains. Most practical enterprise workloads do not need this level of mathematics, but the reasoning skill transfers.

Saturation risk: 🟡 Medium. Hard problems still separate models but frontier models are closing the gap fast.

HellaSwag (Harder Endings, Longer stories, and Situations with Adversarial Generations)

What it measures: Commonsense reasoning — given a story, pick the most plausible continuation from several options. Adversarially generated wrong answers make it harder than the original SWAG dataset.

What it misses: Open-ended generation, temporal reasoning, and any scenario requiring domain expertise. HellaSwag tests if a model can avoid the obviously wrong choice.

Real-world correlation: 🟡 Moderate. Strong HellaSwag scores correlate with general commonsense but not with task-specific reliability. Useful as a floor check.

Saturation risk: 🔴 High. Contamination is widespread. Most frontier models score above 90%.

Long-context

Needle-in-Haystack (NIAH)

What it measures: Can the model find a single deliberately planted fact within a large volume of filler text? Tests basic retrieval within context windows of 4K to 1M+ tokens.

What it misses: Synthesis across multiple document sections, contradiction detection, summarisation, and any task requiring reasoning over dispersed information. NIAH is the “find Waldo” of long-context benchmarks — useful for a single capability, misleading when cited as proof of genuine long-document understanding.

Real-world correlation: 🔴 Weak. A model can score 99% on NIAH at 1M tokens and still fail to summarise a 50-page legal contract or identify conflicting clauses across sections. NIAH success is necessary but not sufficient for real long-context work.

Saturation risk: 🔴 High. Many models score near-perfect on standard NIAH configurations.

See our long-context benchmarks guide.

RULER (Long-Context Multi-Task Benchmark)

What it measures: A suite of long-context tasks beyond simple retrieval: multi-hop QA, variable-tracking, common string extraction, and aggregation over large texts. Tests whether the model can do more than just find a needle.

What it misses: Real document complexity — RULER uses synthetic texts, not actual contracts, reports, or codebases. It tests the model’s raw ability to process long sequences but not its ability to handle real document structure.

Real-world correlation: 🟡 Moderate. Better signal than NIAH alone. A model that performs well on RULER is more likely to handle real long-context workloads, but domain-specific testing is still essential.

Saturation risk: 🟢 Low. Design preserves a gap at the top.

Tool use & agents

BFCL (Berkeley Function Calling Leaderboard)

What it measures: Can the model select the correct function from a list and populate its arguments correctly? Tests tool selection, parameter extraction, and structured output reliability.

What it misses: Multi-turn tool use, error recovery, tool output interpretation, permission boundaries, and the ability to stop or escalate when tools return unexpected results. BFCL scores do not predict whether an agent will run safely in production.

Real-world correlation: 🟡 Moderate. A strong BFCL score means the model can probably handle single-turn tool calls reliably. It says nothing about multi-step agent workflows, loop avoidance, or cost control.

Saturation risk: 🟡 Medium. Advanced models are clustering near the ceiling on simpler categories.

See our function-calling benchmarks guide.

GAIA (General AI Assistants Benchmark)

What it measures: Multi-step agent tasks that require planning, tool use, web search, file parsing, and reasoning. Each task is designed as a real-world assistant request that cannot be solved by a single model call.

What it misses: Safety, cost awareness, and real production constraints (rate limits, API costs, timeouts, multi-user access). GAIA measures raw agentic capability in an ideal environment.

Real-world correlation: 🟢 Good. Currently the best single benchmark for agent readiness. A model that scores well on GAIA is more likely to handle real multi-step tasks than one that only excels at single-turn benchmarks.

Saturation risk: 🟢 Low. Designed to be difficult enough that most models still score below 50%. Significant headroom for improvement.

Human preference & instruction following

Chatbot Arena (LMSYS / lmarena.ai)

What it measures: Blind pairwise human preference — humans chat with two models side by side and pick which response they prefer. Scores are reported as ELO ratings.

What it misses: Task-specific accuracy, cost, latency, safety alignment, and structured output quality. Chatbot Arena measures which model writes more charming, fluent, or helpful prose — not which model is more correct, cheaper, or safer.

Real-world correlation: 🟡 Moderate for user-facing chat. A high Chatbot Arena ELO predicts good conversational experience. It does not predict factual accuracy, code quality, or enterprise reliability.

Saturation risk: 🟡 Medium. ELO gaps of <30 points are often noise. Population drift (who votes, what tasks they try) changes rankings over time.

See our benchmark leaderboards for busy buyers guide.

MT-Bench (Multi-Turn Conversation Benchmark)

What it measures: Multi-turn conversation quality across eight categories (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). Scored by a strong LLM-as-judge (usually GPT-4).

What it misses: Real human preferences, cost, latency, safety, and any dimension the judge model cannot evaluate well. The judge model introduces its own biases and blind spots — it may prefer certain writing styles or penalise appropriate hedging.

Real-world correlation: 🟡 Moderate. Reasonable signal for multi-turn conversational quality but limited by the artefact of LLM-as-judge ratings. Use as one input, not a verdict.

Saturation risk: 🟡 Medium. Top models cluster closely.

AlpacaEval (Instruction Following Benchmark)

What it measures: How well the model follows open-ended instructions compared to a reference (usually GPT-4 or Claude). Uses an LLM-as-judge to determine whether the response is “better” than the reference.

What it misses: Length bias — AlpacaEval strongly rewards longer, more detailed responses, which does not always correlate with better quality. Also inherits the judge model’s stylistic preferences.

Real-world correlation: 🔴 Weak for practical use. Length bias and judge artefacts make AlpacaEval scores noisy and gameable. Use it as a directional signal for instruction-following ability but do not base procurement decisions on it.

Saturation risk: 🔴 High. Most tuned models score above 80% against GPT-4 baselines.

Safety & truthfulness

TruthfulQA

What it measures: Whether a model answers 817 questions truthfully rather than repeating common misconceptions. Questions are designed to elicit false answers that humans commonly believe (e.g., “Is the Great Wall of China visible from space?”).

What it misses: Defensive refusal (saying “I cannot answer” instead of giving the truthful answer), harmful instruction refusal, and safety alignment beyond factual accuracy. TruthfulQA tests one specific failure mode — confident falsehoods that match popular belief.

Real-world correlation: 🟢 Good as a red-team sanity check. Low TruthfulQA scores are a clear warning sign that the model will produce plausible-sounding falsehoods. High scores are table stakes for any production system.

Saturation risk: 🟡 Medium. Safety-tuned models score well but the gap between base and instruct models is still informative.

Safety benchmark suites (SimpleSafetyTests, XSTest, SafetyBench)

What they measure: Whether the model refuses or handles harmful requests appropriately — toxic content, dangerous instructions, privacy violations, and controversial topics.

What they miss: Adversarial attacks, jailbreaks, context manipulation, and long-tail safety edge cases. Safety benchmarks test surface-level refusal but do not predict jailbreak robustness.

Real-world correlation: 🟡 Moderate for basic safety posture. A model that fails basic safety benchmarks should not be deployed. A model that passes them still needs red-teaming for your specific use case.

Saturation risk: 🟡 Medium. Simple safety tests saturate quickly; harder adversarial tests still separate models.

Aggregate & meta benchmarks

HELM (Holistic Evaluation of Language Models — Stanford CRFM)

What it measures: A multi-metric evaluation framework covering accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across dozens of scenarios. Not a single score — a dashboard.

What it misses: Real-world deployment conditions, cost, latency variance, and prompt-template sensitivity. HELM is the most comprehensive public evaluation but still measures model capability, not product fit.

Real-world correlation: 🟢 Good for research-grade assessment. HELM’s multi-metric approach catches weaknesses that single-benchmark evaluations miss. Few teams have the time to replicate this depth internally, so HELM results are a strong starting point.

Saturation risk: 🟢 Low by design. Multi-metric framework adapts as models improve.

LiveBench (Contamination-Limited Fresh Benchmark)

What it measures: Objective, contamination-controlled tasks across reasoning, coding, math, language, and instruction following. Questions are refreshed monthly from recent sources that cannot be in training data.

What it misses: Human preference, conversational quality, and real-world deployment constraints. LiveBench prioritises objectivity and freshness over ecological validity.

Real-world correlation: 🟢 Good. Currently the most trustworthy single benchmark for model comparison because it controls for contamination and uses objective scoring. A model that performs well on LiveBench is genuinely capable, not just memorising.

Saturation risk: 🟢 Low. Monthly refresh maintains separation. Freshness prevents the contamination ceiling that plagues older benchmarks.

How to use this cheat sheet

1. Check which benchmark the claim is based on. If it is HumanEval or AlpacaEval, treat the claim as weak evidence. If it is SWE-bench, GAIA, or LiveBench, treat it as stronger signal.

2. Look for the methodology. A score without evaluation setup details (few-shot vs zero-shot, prompt template, judge model, scaffolding) is less useful. Always check how the number was produced.

3. Cross-reference across categories. A model that dominates knowledge benchmarks but scores poorly on coding or tool use has specific blind spots. A model that dominates everything but is 10x the cost may still be the wrong choice.

4. Run your own eval. No benchmark replaces testing on your prompts, your documents, and your risk tolerance. Use this cheat sheet to shortlist models, then run a small workload-specific evaluation before committing.

5. Watch for saturation. If every model scores above 90%, stop treating score differences as meaningful on that benchmark. Move to harder or fresher benchmarks for differentiation.

Related guides

• How LLM benchmarks work, and what they miss — deeper dive into methodology, contamination, and saturation
• Benchmark leaderboards for busy buyers — how to read Chatbot Arena, LiveBench, and when to ignore rankings
• Coding benchmarks explained — HumanEval, MBPP, SWE-bench in detail
• Long-context benchmarks — NIAH, RULER, and real document understanding

Methodology

• Data checked: 2026-06-08
• Sources consulted: Benchmark papers and documentation — MMLU-Pro (arXiv), GPQA (arXiv), SimpleQA (OpenAI), HumanEval (OpenAI GitHub), MBPP (Google Research), SWE-bench (Princeton, swebench.com), GSM8K (Cobbe et al., OpenAI), MATH (Hendrycks et al., OpenAI), HellaSwag (AllenAI), Needle-in-Haystack (Kamradt), RULER (Hsieh et al., arXiv), BFCL (Berkeley GitHub), GAIA (Meta/FAIR, Hugging Face), Chatbot Arena (LMSYS, lmarena.ai), MT-Bench (LMSYS), AlpacaEval (Stanford/tatsu-lab), TruthfulQA (Lin et al.), HELM (Stanford CRFM), LiveBench (livebench.ai), Cleanlab contamination survey (2024)
• Assumptions: Real-world correlation ratings are editorial judgement based on documented benchmark limitations and industry consensus, not a formal meta-analysis. Saturation assessments reflect publicly reported scores as of the check date. The cheat sheet covers the 18 most-cited benchmarks; smaller or niche benchmarks are excluded.
• Limitations: This cheat sheet does not include current model scores (which change weekly). It is a methodology reference, not a ranking. Real-world correlation ratings are directional — your workload may differ. Domain-specific benchmarks (medical, legal, financial) are not covered. The article assumes the reader has basic familiarity with LLM evaluation concepts.
• Jurisdiction: Global. Benchmark methodology is jurisdiction-agnostic. Regulatory references (EU AI Act) are noted where relevant but this is not regulatory advice.

Source list

• MMLU-Pro paper — https://arxiv.org/abs/2406.00368 (accessed 2026-06-08)
• GPQA paper — https://arxiv.org/abs/2311.12022 (accessed 2026-06-08)
• SimpleQA — https://openai.com/index/introducing-simpleqa/ (accessed 2026-06-08)
• HumanEval — https://github.com/openai/human-eval (accessed 2026-06-08)
• MBPP — https://github.com/google-research/google-research/tree/master/mbpp (accessed 2026-06-08)
• SWE-bench — https://www.swebench.com/ (accessed 2026-06-08)
• GSM8K — https://arxiv.org/abs/2110.14168 (accessed 2026-06-08)
• MATH — https://arxiv.org/abs/2103.03874 (accessed 2026-06-08)
• HellaSwag — https://arxiv.org/abs/1905.07830 (accessed 2026-06-08)
• RULER — https://arxiv.org/abs/2404.06654 (accessed 2026-06-08)
• BFCL — https://gorilla.cs.berkeley.edu/leaderboard.html (accessed 2026-06-08)
• GAIA — https://huggingface.co/gaia-benchmark (accessed 2026-06-08)
• Chatbot Arena — https://lmarena.ai/ (accessed 2026-06-08)
• MT-Bench — https://arxiv.org/abs/2306.05685 (accessed 2026-06-08)
• AlpacaEval — https://tatsu-lab.github.io/alpaca_eval/ (accessed 2026-06-08)
• TruthfulQA — https://arxiv.org/abs/2109.07958 (accessed 2026-06-08)
• HELM — https://crfm.stanford.edu/helm/latest/ (accessed 2026-06-08)
• LiveBench — https://livebench.ai/ (accessed 2026-06-08)

Trust Stack

• Last checked: 2026-06-08
• Corrections: Contact us to report errors

Change log

• 2026-06-08: Editorial review — added Quick Answer, 4 Editor’s Notes, slugified H2/H3 IDs, Methodology, Source List, Trust Stack, Change Log; updated frontmatter with correct review model label and check date; renamed Related guides section
• 2026-06-07: first published (draft)