Hot AI News

Elo ratings for AI models: How blind human votes rank chatbots

Elo-style scores rank AI models by asking people which of two anonymous answers is better. A plain-language guide to how the numbers work, who uses them, and what they cannot tell you.

Elo ratings for AI models: How blind human votes rank chatbots
Photo illustration: Aiholics

An Elo-style rating for an AI model is a score built from head-to-head votes. People see two anonymous answers to the same prompt, pick the better one, and a formula turns thousands of those picks into a ranking. Higher means people preferred the model more often.

The idea is simple, but the numbers in model launches are easy to misread. This guide explains what a gap of 47 points really means.

A chess ladder for chatbots

Picture a chess club ladder. Beat a strong player and your rating jumps. Beat a weak one and it barely moves. After enough games, ratings predict who will win a future match.

Now swap the players for AI models. A visitor types any question, gets two answers from two hidden models, and votes for the one they like more. After voting, the names are revealed. Each vote is one game, and the models climb or fall like chess players.

Nobody has to write a grading rubric. The judges are ordinary users, and they compare answers on tasks they care about.

Advertisements

How votes turn into a number

A person selects one of two anonymous AI answers on a touchscreen with paired rating indicators below.
Photo illustration: Aiholics

The classic Elo system, designed for chess, gives each player a score. The gap between two scores predicts the odds of a win. On the standard scale, a 400-point gap means the stronger side is expected to win about 91% of the time. A smaller gap means a closer contest.

That lets readers translate headlines. In our weekly roundup, OpenAI reported that GPT-6.2 Astra scored 1,318 on a blind vote, against 1,271 for Claude Fable 5.2 and 1,259 for GPT-6.2 Sol (see New AI models this week). If that vote follows the standard Elo scale, a 47-point lead means roughly a 57% expected win rate over Fable 5.2. That is an edge, not a blowout.

Those figures are OpenAI’s own report of a third-party vote, so they should be read as the maker’s claim.

Where the method came from

The best-known version is Chatbot Arena, described in a 2024 paper by Wei-Lin Chiang, Lianmin Zheng, Ion Stoica and colleagues, mostly at UC Berkeley. The paper, titled “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference,” was posted to arXiv on March 7, 2024. The team ran it as a public website where users type their own prompts, with no preset questions.

The paper reports more than 240,000 votes from over 90,000 users across more than 50 models, as of January 2024. About 77% of the prompts were in English. The designers filtered out names such as “GPT” and “Claude” from answers so users could not tell which model they were judging, and they allowed ties and a “both are bad” vote.

Why the math is not quite Elo

Despite the name people still use, the Chatbot Arena authors say they moved from online Elo updates to the Bradley-Terry model. Their reason: the Bradley-Terry coefficients are better for statistical estimation. Bradley-Terry fits one score per model to all votes at once, rather than updating after each game.

This matters because order no longer affects the result. Elo updates depend on the sequence of games. A fit over all votes does not. The paper also reports confidence intervals, so two models whose intervals overlap should be treated as roughly tied.

The team also sampled model pairs unevenly, putting more votes on matchups with similar strength. That gives tighter rankings for the same number of votes.

The paper checked quality by having graduate-student experts relabel 160 battles. It reports agreement between 72% and 83% across crowd, expert and GPT-4 judge labels. Crowd votes are useful, but they are not a perfect stand-in for expert judgment.

Advertisements

Who relies on these scores today

Arena-style leaderboards are now a standard marketing number. Labs quote them in launch posts, and our weekly model roundup carries one for Astra and Sol. Other benchmarks work differently. For example, an intelligence index and cost per task measure scored test results, not preference.

The vote-based approach fits tasks that have no single right answer, such as writing, chat and explanation. It is less suited to checking whether code runs or a math answer is right, where an automatic check is cheaper and more objective.

What the score cannot tell you

Votes reward what people like, and people like some things that are not correctness. A style-control study from the same Arena team (2024) added answer length and markdown formatting, such as headers, bold and lists, as variables in the rating model. It reported length as the dominant style factor. A model can rise partly by writing longer, neater answers.

Other limits are worth keeping in mind:

  • Who votes. Users choose themselves, and the paper’s audience skews toward hobbyists and researchers. Their prompts may not match yours.
  • One number hides categories. The paper shows win rates between the same two models swinging widely by topic, from above 96% on some coding prompts to under 60% elsewhere.
  • Helpfulness, not safety. Votes mostly reflect which answer felt better, not which was safer or more factual.
  • Self-reported figures. A lab quoting a leaderboard result may choose the category or settings that look best.

What to check next time a launch quotes an Elo score

Four questions help. Who ran the vote, and was it blind? Does the score control for answer length and style? How large is the confidence interval compared with the gap? And does the category match your own use?

A 47-point lead in general chat may vanish on your coding or document task. The most trustworthy move is to try two models on your own prompts and treat the leaderboard as a starting point.

Expect the method to keep evolving. Style controls, category-specific boards and agent tasks that run for hours are all harder to judge with a single quick vote, and each is an open area for the evaluation community.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like