Near-copy AI models split Chatbot Arena votes more decisively than ratings predict

Key points

• On 1.8 million Arena votes, versions of one model were 1.37 times as decisive as Bradley-Terry predicts.
• Versions from one lab, across different generations, showed smaller multipliers of 1.26 and 1.12.
• The leaderboard order barely moves, and the authors say better head-to-head odds are not established.
• Near-copies take votes mostly from their twin when pooled over prompts; the effect is smaller within a prompt.

Why it matters for marketers: Content teams relying on Arena rankings should note that closely related models may be separated by ties and voter patterns, not by reliably better head-to-head odds.