I Stopped Trusting AI Benchmarks After One Bad Week

For a while I picked models by the leaderboard, and then a model that topped three charts wrote me a confidently wrong paragraph about a tax rule. The benchmark said it was smart. The paragraph said it had no idea what it was talking about.

Benchmarks measure a narrow, clean slice of work. Real tasks are messy, underspecified, and full of the context the test didn't include. A model that aces a reasoning benchmark can still invent a citation on your specific question.

What I trust now is the number that comes from my own use: how often do I have to re-check its output before I'd put my name on it. That's a benchmark you build by doing the work, not by reading a chart someone published.

The labs know this. That's why they ship adjustable thinking effort and routing models: they're admitting the average score hides the part that matters, which is whether it's right on your job.

Read the benchmarks for the trend, not the verdict. Then run your own thirty-example test on the thing you actually do, and let that be the number you remember.