Perplexity: How a language model’s guessing score became a capability yardstick
Perplexity scores how well a language model predicts the next word. Lower means less surprised. Here is how it works, what it hides, and why OpenAI now ties capability limits to it.

Perplexity is a score for how well a language model predicts text. The less surprised the model is by what comes next, the lower its perplexity. It is the standard measure of how well a model has learned language, and OpenAI recently put it at the center of a safety proposal.
Do not confuse it with the search company Perplexity. This is an older, general idea in language modeling.
A guessing game with a number attached
Imagine finishing the sentence “The cat sat on the …”. A good guesser narrows it to a few likely words, such as “mat” or “sofa”. A poor guesser sees thousands of equally plausible options.
Perplexity turns that into a number. Roughly, it is how many equally likely choices the model is torn between at each step. A perplexity of 3 means the model behaves as if it were picking among three options each time. A perfect model scores 1. A model that treats every word in a vocabulary of V words as equally likely scores V, as Sebastian Raschka’s explainer notes.
From probabilities to one score

A language model reads text and, at each position, assigns a probability to every possible next token. A token is a chunk of text, often a word or part of one. After the real next token appears, you look at the probability the model gave it. High probability means little surprise. Low probability means a lot.
Average the surprise over a large body of text and you get the cross-entropy, the same loss used to train the model. Perplexity is the exponential of that loss. The Hugging Face documentation states exactly this relationship: perplexity equals the exponentiation of the cross-entropy between the data and the model’s predictions.
So the two are the same information on different scales. Labs often report the loss, or its logarithmic form, because it is easier to plot and compare over time. Lower is better in both cases.
Why OpenAI is now publishing it
On November 22, 2026, OpenAI chief scientist Jakub Pachocki published the log-perplexity of OpenAI’s models and proposed that labs agree on shared limits tied to it. We covered the details in our news report.
According to that post, GPT-3.5 scores about 0.99 and GPT-4 about 0.91 on OpenAI’s main evaluation dataset. OpenAI links models below 0.8 to emerging abilities in cybersecurity and mathematics, and a model near 0.76 to major results on open math problems.
A rough sense of scale helps. If the figure is a natural-log loss, the exponential turns 0.91 into about 2.5 and 0.76 into about 2.1. The model is then about as uncertain as a choice between two or three options per step. That conversion is our own arithmetic. The units and normalization (per token or per byte, for example) are not spelled out in the material we reviewed, so treat it as an illustration only.
The proposal itself is a pacing mechanism. Developers would commit not to cross a perplexity limit until tools for understanding and monitoring AI behavior reach an agreed milestone. The idea is that perplexity is a number every lab can compute, which makes it checkable in a way that vague descriptions of capability are not.
Why a lower score does not simply mean smarter
Perplexity depends heavily on what you measure it on. A model can look excellent on one text collection and mediocre on another. OpenAI’s own post, as we reported, says labs use different datasets and data mixtures, which makes cross-lab comparison hard. The figures it gave come from one held-out set built from an internal codebase.
Tokenization matters too. The Hugging Face documentation notes that models with different tokenizers are not directly comparable, because they cut text into different pieces and therefore predict different units. It also says the metric suits classical autoregressive models, which predict the next token, and is not well defined for masked models such as BERT.
The post lists further limits. Perplexity does not directly correspond to a task people care about. It ignores gains that come from outside pretraining: chain-of-thought reasoning, where a model works through steps before answering, does not lower perplexity yet can raise capability a great deal. And the floor is unknown, because the true entropy of the data cannot be measured.
What to watch next
A shared limit only works if rivals measure the same way. Three things are still open, according to our reporting. No common evaluation dataset has been named. No value has been agreed for the first limit. No other lab has committed to the idea.
The figures are also OpenAI’s own and have not been independently checked. A useful sign would be other labs publishing their own numbers on a common held-out set, and outside researchers confirming that perplexity tracks the abilities OpenAI describes.
Until then, perplexity is best read as a precise but narrow gauge. It tells you how well a model predicts text. It does not by itself say what the model can do.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…
You may also like

Perceptual hashing: How platforms spot known abuse images without viewing them

Content Security Policy: How a browser rule limits injected code, and where it fails

Monocular metric depth: How one photo can tell a model distances in meters
