The Benchmark Industrial Complex Is Lying to You
Every model "beats GPT on this benchmark." Almost none of those numbers mean what the tweet implies.
A benchmark is a test someone wrote. The moment a model is trained to ace it, the test is no longer measuring what you think it measures. This is not a conspiracy. It is just how measurement works.
I have lost count of announcements where a model "surpasses human performance" on task X, and task X turns out to be a multiple-choice quiz with a known answer key the model may have seen.
The honest benchmarks are the boring ones: does it answer my actual support tickets correctly, on my actual data, most of the time. Nobody tweets that because it is not a number you can put on a leaderboard.
When you evaluate a tool, ignore the leaderboard. Run it on ten real things you do. The scoreboard is for fundraising. Your inbox is for truth.
None of this means the models are bad. They are good. They are just not as good as their own press releases, and not as bad as their critics claim.