Hot AI News

OpenAI publishes its models’ perplexity and proposes shared limits on it

Chief scientist Jakub Pachocki shared OpenAI's perplexity numbers from GPT-3.5 onward and asked labs to agree on common limits tied to that metric.

OpenAI publishes its models’ perplexity and proposes shared limits on it
Image: OpenAI

OpenAI chief scientist Jakub Pachocki has published the perplexity of OpenAI’s models, a measure of how well they predict data, and proposed that labs agree on common limits tied to it. He says “as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements.”

The numbers OpenAI put on the table

In the post, “Perplexity and Intelligence”, dated November 22, 2026, Pachocki reports log-perplexity, the cross-entropy loss OpenAI says it usually tracks. On its main evaluation dataset, GPT-3.5 scores about 0.99 and GPT-4 about 0.91. Lower is better.

OpenAI shares its models' perplexity, or how well they predict data, and proposes pacing limits on such metrics, citing a possible 2027 "avalanche" of advances (Jakub Pachocki/OpenAI)
Image: Techmeme

The post links these values to capability. Models below 0.8 are tied to emerging transformative abilities in cybersecurity and mathematics. A model at about 0.76 is tied to what OpenAI describes as major results on open math problems. The post also points to a window of roughly 0.68 to 0.7, where it expects further transformative capabilities.

OpenAI shares its models' perplexity, or how well they predict data, and proposes pacing limits on such metrics, citing a possible 2027 "avalanche" of advances (Jakub Pachocki/OpenAI)
Image: Techmeme

The main held-out dataset since GPT-3.5 comes from OpenAI’s internal codebase, using a fixed snapshot. The post does not name any other dataset for these figures.

Advertisements

A proposal for shared limits, with a catch

The core proposal is an industry standard with three parts: a first limit, a shared metric to track it, and a milestone that must be reached before going further. Developers would commit not to cross the limit until the ability to understand and monitor AI behavior reaches that milestone. The post’s example is a successor to chain-of-thought monitors. It also asks frontier developers to agree on shared evaluation norms, red-team safety cases and transparency standards.

OpenAI says it expects AI to help with alignment and monitoring, but that progress there will probably not keep pace with the pretraining acceleration. That mismatch is the stated reason for pacing limits.

Perplexity is a rough yardstick, and the post says so

The post lists four weaknesses. Labs use different evaluation datasets and data mixtures, so cross-lab comparison is hard. Perplexity is detached from actionable evaluations; in the post’s words, it “does not directly correspond to a task we care about.” It ignores gains from outside pretraining: chain-of-thought reasoning does not lower perplexity yet greatly raises capability. The fourth is that the floor is unknown, since the entropy of the data is not known.

These caveats matter for any shared limit. A threshold only binds if rivals measure the same way, and the post gives no common dataset, no agreed value for the first limit and no commitment from other labs. The figures are OpenAI’s own and have not been independently checked. The post does not say which math results the 0.76 model produced. Separately, an Anthropic mathematician’s claim that Claude helped prove the Riemann hypothesis was reported by us on November 21 and also remains a claim. Neither lab’s math results can be matched against a loss value from public data.

The post is also a proposal, not a policy. OpenAI is not announcing that it will halt any training run. Startuphub’s summary of the post describes the idea as a pause, but that wording is the outlet’s, not OpenAI’s.

Advertisements

What readers can use today

Nothing here is a product or a release. The practical value is the disclosure: until now, outsiders had no OpenAI loss values from GPT-3.5 to the latest models. Researchers can now reference the 0.99, 0.91 and 0.76 figures and the 0.68 to 0.7 window, with the caveat that the dataset is OpenAI’s internal code and cannot be reproduced. Whether Anthropic, Google or others publish comparable numbers will decide if the idea becomes a real standard.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Alex CarterAI News & Big Tech Correspondent

Alex Carter writes for Aiholics, keeping readers updated on the fast-paced world of AI and Big Tech.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like