Researchers Show Arena-Style AI Leaderboards Can Be Gamed for Rank Gains of Up to 45 Places

A new arXiv paper shows that Arena-style leaderboards, which rank language models through anonymous pairwise votes aggregated by the Bradley-Terry model, can be manipulated. The authors identify three tactics: strategic voting, strategic prompting, and strategic nomination of irrelevant models. Tested on public leaderboard data, each attack boosted a model by up to 15-20 places, and combined they reached 45 places. The authors also describe safeguards and note the flaw applies to any open preference-based evaluation using Bradley-Terry.

Why it matters: Teams that choose or market models based on public preference leaderboards need to know those rankings can be inflated and should weigh them with other evaluations.