Every model launch comes with a chart showing it on top. They can't all be right — and they mostly aren't lying, either. Reading benchmark claims well is a core skill for anyone choosing a model. Here's the field guide.
Why the charts always favor the new model
Labs pick the benchmarks and settings that flatter their model — not usually by faking numbers, but by choosing the tests, configurations, and comparison points where they win. A model can genuinely top three benchmarks and trail on five others you weren't shown. The chart is true and incomplete at the same time.
"State of the art" almost always means "on the benchmarks we chose, in the configuration we chose."
The questions to ask
- Which benchmark, and does it match my task? Topping a math benchmark says little about your customer-support workload.
- What settings? Effort level, prompting, and tools can swing scores enormously. Were the comparisons run fairly?
- Contamination? If a benchmark's answers leaked into training data, high scores mean memorization, not capability.
- Who ran it? Self-reported numbers deserve more caution than independent evaluations.
The only test that counts
The uncomfortable truth: no public benchmark predicts performance on your specific task well enough to trust blindly. Build a small evaluation set from your own real cases and run the candidate models on it. A quick, honest eval on twenty of your actual examples tells you more than any leaderboard.
This is the "signal, not noise" habit in practice: treat benchmark claims as a starting hypothesis, verify on your own data, and never mistake a launch chart for a verdict.
