Benchmark Scores Are Better Than the Sceptics Say and Worse Than the Vendors Claim
The evaluations are real, the methodology is interesting, and the gaps between models are smaller than the charts suggest.
Every model release arrives with a bar chart, and the bars are always close together. Two points here, four points there. The charts are not fabricated. They are also close to useless for deciding what to actually ship.
What the benchmarks get right
Standardised evaluation was a genuine advance. Before it, model comparison was vibes and cherry-picked transcripts. A shared test set with a published methodology means claims can be checked by someone with an incentive to disprove them, and that is most of what makes a field self-correcting.
The benchmarks also catch real regressions. A model that drops ten points on a reasoning suite has usually lost something you will notice.
Where the numbers stop meaning what you think
The first problem is contamination. Test sets are published, published text ends up in training corpora, and a model can score well on questions it effectively memorised. This is not usually deliberate. It is very hard to avoid at web scale, and it inflates exactly the benchmarks that are most cited because those are the ones most discussed online.
The second is that the gaps are frequently inside the noise. Many suites have a few hundred to a few thousand items. A two point difference on a thousand item test is roughly twenty questions. Rerun with a different prompt template, a different temperature, or a different answer parser, and twenty questions move.
The third, and the one that matters most in practice: benchmarks measure single-turn answers to well-posed questions. Almost nothing anyone builds is single-turn answers to well-posed questions. It is long context, ambiguous instructions, tool calls, retries, and structured output that has to parse.
The evaluation that actually predicts anything
Build a test set out of your own traffic. A hundred real cases beats a public benchmark of ten thousand, because it measures the distribution you are actually serving instead of the one the leaderboard is serving.
Pin the failures. For each case, write what a wrong answer looks like specifically, not just what a right one looks like. Assertions that only check for success pass for the wrong reasons far more often than anyone expects.
And measure the whole pipeline. Your users experience retrieval, prompt, model, parser and fallback as one system. Swapping the model and measuring the model tells you very little about what they will feel.
What to take from the charts
Read them as a coarse filter, not a ranking. They will reliably tell you which models are in the same class. They will not tell you which one is better at your problem, and a two point gap is not a reason to migrate anything.