OpenAI Reveals Its Models' Next-Word Prediction Error: From 0.99 for GPT-3.5 to Below 0.8
The better a model predicts the next word in a text, the "smarter" it is. That is perplexity, and the lower the number, the better. OpenAI's chief scientist Jakub Pachocki proposes that all labs compare models this way. The log perplexity of GPT-3.5 is about 0.99, GPT-4 is about 0.91, and current models are below 0.8. Strong cybersecurity and math skills appeared from that level onward. For this reason, base training deserves at least as much attention as benchmarks.
AI-processed from OpenAI Blog; edited by Hamidun News
Picture autocomplete on a phone: it is more useful the more often it guesses which word you will type next. Today's large models are built on the same principle, except they learn from enormous volumes of text and code. OpenAI's chief scientist Jakub Pachocki has published an essay proposing that AI progress be measured precisely by the quality of this guessing. The metric is called perplexity, and OpenAI has disclosed its internal figures for it for the first time, calling on other labs to do the same.
What perplexity is, in a simple example
Suppose a model sees the start of the code line def add(a, b): return a and must predict the continuation. If it almost always produces + b, its prediction error is low. If it guesses at random, the error is high. Perplexity averages this error across a huge set of texts the model did not see during training. In practice, OpenAI looks at the log of perplexity, also known as cross-entropy. These are not accuracy percentages: the lower the number, the better the model predicts.
Pachocki calls a trained model a "compressor." To predict new data well, it must capture the patterns in that data. But it is not an archive like GZIP: the model does not have to remember the corpus itself. Everything else, including prompting, fine-tuning, and reinforcement learning, only draws out hidden abilities from this foundation. Which specific abilities will emerge is unclear in advance.
OpenAI's figures on its main validation set:
- GPT-3.5, about 0.99: coherent dialogue, the basis of the first ChatGPT;
- GPT-4, about 0.91: useful chains of reasoning, from which reasoning models grew;
- below 0.8: notable skills in cybersecurity and math;
- about 0.76: the model began producing breakthrough results on open math problems.
Pachocki recalls how solutions to questions he had long been familiar with appeared one after another in a single Slack thread, and calls this "surreal." He stresses that no one knew in advance that skills would appear at these particular thresholds. Earlier still, GPT-2 learned to write coherent paragraphs, and GPT-3 learned from a few examples given right within the prompt.
Why limits, and what comes next
The author names the next window as roughly 0.68–0.70. He openly admits he does not know what abilities will emerge there or whether we will even be able to test for them. He also expects that the usual alignment audits will work worse, and that models will get better at recognizing that they are being tested.
Therefore, OpenAI proposes an industry standard. Labs would agree in advance on a first perplexity limit and other measurements, and commit not to cross it until they reach a specific milestone in understanding and monitoring AI behavior. One example of such a milestone is a successor to chain-of-thought (CoT) monitoring. Subsequent limits could then be set on the same principle. Common evaluation norms and red-team safety checks are also needed by the time a model first launches.
There is also a forecast about pace. Pachocki believes AI could dramatically accelerate algorithmic research on pretraining, and that a "avalanche" of fundamental discoveries is possible as early as 2027, as has already happened in mathematics. Progress in alignment and monitoring, in his assessment, is unlikely to keep up with such acceleration. The chart shows a sharp inflection in 2026. OpenAI explains it mainly through long-term research and investment in compute, with automation playing only a partial role. But AI already contributes to the design of recent models.
Limitations of the metric
Pachocki himself lists the drawbacks:
- Labs use different validation sets and data mixtures, so direct comparison is not yet possible;
- OpenAI calculates perplexity on a sample from its internal code, and for GPT-2 and GPT-3 such a measurement poorly reflects general capabilities;
- perplexity does not correspond to any particular applied task, and chain-of-thought reasoning does not lower it, even though it sharply strengthens the model;
- the lower bound of the data itself is unknown: zero cannot be reached, and as the score nears the floor, progress on the scale will slow, even though each small fraction will mean a bigger jump in intelligence.
What this means in practice
For developers and product managers, the takeaway is this: benchmarks saturate quickly, and most older tests were "solved" at roughly the GPT-4 level. The pretraining metric remains a more stable reference point, and OpenAI proposes making it a common language for the industry. When a lab announces a new base model, it makes sense to ask not only about benchmark scores but also about the prediction error on an open dataset.
Whether the metric becomes a standard depends on competitors. Publishing internal figures gives them a clear picture of OpenAI's position, and the company knowingly took on this risk in pursuit of shared safety standards.
- Nov 22, 2026Ahrefs: A Brand With 10x the Web Mentions Gets Only 3.7x More in AI Answers
- Nov 22, 2026Axiom Math's AI Writes 43.1 Million Lines of Lean Proof in 24 Days
- Nov 22, 2026Yandex Grant Funds an LLM Engine: 35B Model on RTX 2080 Ti Runs Three Times Faster Than llama.cpp
- Nov 22, 2026OpenAI's New Astra Model Writes 40% Shorter, but Its Hedging Score Fell from 96 to 24
Key Questions Answered
What perplexity is, in a simple example?
Suppose a model sees the start of the code line def add(a, b): return a and must predict the continuation. If it almost always produces + b, its prediction error is low. If it guesses at random, the error is high. Perplexity averages this error across a huge set of texts the model did not see during training. In practice, OpenAI looks at the log of perplexity, also known as cross-entropy. These are not accuracy percentages: the lower the number, the better the model predicts.
Why limits, and what comes next?
The author names the next window as roughly 0.68–0.70. He openly admits he does not know what abilities will emerge there or whether we will even be able to test for them. He also expects that the usual alignment audits will work worse, and that models will get better at recognizing that they are being tested. Therefore, OpenAI proposes an industry standard.
What are the key details on Limitations of the metric?
Pachocki himself lists the drawbacks: Labs use different validation sets and data mixtures, so direct comparison is not yet possible; OpenAI calculates perplexity on a sample from its internal code, and for GPT-2 and GPT-3 such a measurement poorly reflects general capabilities; perplexity does not correspond to any particular applied task, and chain-of-thought reasoning does not lower it, even though it sharply strengthens the model; the lower bound of the data itself is unknown: zero cannot be…
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.