Choosing a model when benchmarks can't keep up
TL;DR
By mid-2026, the reference LLM leaderboards have frozen or pivoted, and what stays current is either self-reported by vendors or recopied by unverified aggregators. One independent harness still holds — and it saturates at the top. Result: the source of truth left is your own codebase. Here's how I decide without leaning on numbers nobody can verify anymore.
I wanted to compare eight open models to decide which to put on my layering. I went looking for independent, up-to-date benchmarks. There’s barely one left.
Where did the reference leaderboards go?
Not “all dead” — but the ones you could lean on have frozen or changed jobs. HAL, Princeton’s leaderboard, archived its harness and pivoted to measuring reliability rather than raw performance. Aider’s official leaderboard hasn’t moved since November 2025. Official Terminal-Bench holds none of the models I wanted to test.
Others still run — LiveCodeBench even shipped a v2 in 2026, SWE-bench stays active — but their public rankings mix different harnesses and non-reproducible submissions. On the most-viewed SWE-bench board, nearly every entry is self-reported by vendors, not independently verified. Quoting a model’s “79%” next to another’s “97%” is meaningless: different methods and dates. It’s marketing with a decimal point.
What’s left are aggregators — llm-stats, benchlm — showing fresh numbers. But they recopy without checking: llm-stats itself admits it has no verified results, and benchlm attributes scores to the wrong model. “Current” doesn’t mean “reliable” there.
The only living judge saturates
There’s still vals.ai: an independent harness, still updated, covering most recent models. It’s valuable, and I use it. But you have to read what it actually says: on SWE-bench Verified, the top five fit within four points, with two points of standard error. In other words, the top of the ranking can’t be ordered statistically. The benchmark no longer tells you “this one beats that one,” it tells you “these five play in the same zone.” Useful to eliminate the bottom. Useless to rank the top.
Benchmaxxing isn’t a theory
When you compare the score a vendor announces to the one an independent third party measures, the gap almost always runs the same way. DeepSeek V4 Pro: 80.6% announced, 74% measured by CAISI/NIST — six and a half points lower. Hunyuan 3: 53.2 announced, 31.6% measured by Artificial Analysis — twenty-one points lower. One exception in my sample: Kimi, whose number holds within 0.8 points when verified. One honest vendor out of six — that’s the base rate to keep in mind before believing a slide.
The criterion that matters is measured nowhere
The worst part is that even a perfect code benchmark wouldn’t answer my real question. A CLI agent does nothing but tool-calling: read, edit, run bash, loop back. A model that writes beautiful code but botches one tool call in five is unusable — and no quality leaderboard measures that. Which one actually holds, and for which role, is the subject of the guide; here I stop at the observation.
I measured it the dumb way: the same trivial task, replayed three times on the Go models. Roughly one run in three comes out wrong — not slow, not imperfect: wrong. Yet a single sample gave a clean “it works.” That’s exactly what a single-pass benchmark will never catch, and what matters most when you hand a tool chain to a non-deterministic model.
The verdict that covers them all, from a practitioner who pushed every one: “nothing seemingly comes close to GPT 5.5/5.6 xhigh for tasks beyond 5-10 minutes of work, they all more or less collapse after a while.” Translated: these models hold on a tightly scoped task and collapse over time. No benchmark teaches you that — usage does.
That ceiling lands on a habit I had already measured, and that’s what makes it bearable: across 267 tracked sessions, 62% last under 10 minutes. The limit you suffer on the model side is already, on the usage side, the dominant pattern. It doesn’t make these models good over long runs — it means long runs are rarely what you ask of them.
What I do instead
I stopped waiting for a neutral number that no longer exists. The only source of truth left is my own codebase — and nobody publishes a PHP/Symfony benchmark anyway. So I measure myself: a real tool chain on a real task, a few cents of quota, and it settles what the public sources disagree on. A five-minute run on my stack tells me more than any frozen leaderboard.
It’s not very satisfying, and it doesn’t scale into a clean comparison table. But it’s the only thing that stays true. Benchmarks used to tell you which model to pick; now they only tell you which to eliminate. The final choice happens at home, on your code — signed, dated, measured.