Benchmarking AI image models is more broken than people admit
Leaderboards decide which model you try. Most of them measure the wrong thing, and the models are trained to win them.
Every month a new image benchmark crowns a winner, and every month the winner is suspiciously good at exactly that benchmark. That is not a coincidence; models are tuned to the test.
The metrics that get published - fidelity scores, preference rates - rarely map to what I need: does the text on the image render, does the hand have five fingers, will it match my brand. Those are awkward to score, so they get ignored.
Worse, human preference panels drift toward 'looks impressive' over 'does the job.' A flashy render beats a correct one in a vote, and the leaderboard follows the vote.
I now test image models on my own three-task checklist instead of reading the board. It is slower, but it tells me what I will actually ship.