Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents (Nov 23, 18:30 MSK): 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseResearch Paper · For R&DOpenAI Blog→ original

OpenAI Reveals Its Models' Next-Word Prediction Error: From 0.99 for GPT-3.5 to Below 0.8

The better a model predicts the next word in a text, the "smarter" it is. That is perplexity, and the lower the number, the better. OpenAI's chief scientist Jakub Pachocki proposes that all labs compare models this way. The log perplexity of GPT-3.5 is about 0.99, GPT-4 is about 0.91, and current models are below 0.8. Strong cybersecurity and math skills appeared from that level onward. For this reason, base training deserves at least as much attention as benchmarks.

AI-processed from OpenAI Blog; edited by Hamidun News
OpenAI Reveals Its Models' Next-Word Prediction Error: From 0.99 for GPT-3.5 to Below 0.8
Source: OpenAI Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceOpenAI Blog ↗
Editor-in-Chief
Key metric #1The measurements were made on OpenAI's internal code, which makes comparison with other models difficult
Key metric #2A model with a score of about 0.76 began solving open math problems
Key metric #3Development limits will be lifted once a new method for monitoring AI appears

Picture autocomplete on a phone: it is more useful the more often it guesses which word you will type next. Today's large models are built on the same principle, except they learn from enormous volumes of text and code. OpenAI's chief scientist Jakub Pachocki has published an essay proposing that AI progress be measured precisely by the quality of this guessing. The metric is called perplexity, and OpenAI has disclosed its internal figures for it for the first time, calling on other labs to do the same.

What perplexity is, in a simple example

Suppose a model sees the start of the code line def add(a, b): return a and must predict the continuation. If it almost always produces + b, its prediction error is low. If it guesses at random, the error is high. Perplexity averages this error across a huge set of texts the model did not see during training. In practice, OpenAI looks at the log of perplexity, also known as cross-entropy. These are not accuracy percentages: the lower the number, the better the model predicts.

Pachocki calls a trained model a "compressor." To predict new data well, it must capture the patterns in that data. But it is not an archive like GZIP: the model does not have to remember the corpus itself. Everything else, including prompting, fine-tuning, and reinforcement learning, only draws out hidden abilities from this foundation. Which specific abilities will emerge is unclear in advance.

OpenAI's figures on its main validation set:

  • GPT-3.5, about 0.99: coherent dialogue, the basis of the first ChatGPT;
  • GPT-4, about 0.91: useful chains of reasoning, from which reasoning models grew;
  • below 0.8: notable skills in cybersecurity and math;
  • about 0.76: the model began producing breakthrough results on open math problems.

Pachocki recalls how solutions to questions he had long been familiar with appeared one after another in a single Slack thread, and calls this "surreal." He stresses that no one knew in advance that skills would appear at these particular thresholds. Earlier still, GPT-2 learned to write coherent paragraphs, and GPT-3 learned from a few examples given right within the prompt.

Why limits, and what comes next

The author names the next window as roughly 0.68–0.70. He openly admits he does not know what abilities will emerge there or whether we will even be able to test for them. He also expects that the usual alignment audits will work worse, and that models will get better at recognizing that they are being tested.

Therefore, OpenAI proposes an industry standard. Labs would agree in advance on a first perplexity limit and other measurements, and commit not to cross it until they reach a specific milestone in understanding and monitoring AI behavior. One example of such a milestone is a successor to chain-of-thought (CoT) monitoring. Subsequent limits could then be set on the same principle. Common evaluation norms and red-team safety checks are also needed by the time a model first launches.

There is also a forecast about pace. Pachocki believes AI could dramatically accelerate algorithmic research on pretraining, and that a "avalanche" of fundamental discoveries is possible as early as 2027, as has already happened in mathematics. Progress in alignment and monitoring, in his assessment, is unlikely to keep up with such acceleration. The chart shows a sharp inflection in 2026. OpenAI explains it mainly through long-term research and investment in compute, with automation playing only a partial role. But AI already contributes to the design of recent models.

Limitations of the metric

Pachocki himself lists the drawbacks:

  • Labs use different validation sets and data mixtures, so direct comparison is not yet possible;
  • OpenAI calculates perplexity on a sample from its internal code, and for GPT-2 and GPT-3 such a measurement poorly reflects general capabilities;
  • perplexity does not correspond to any particular applied task, and chain-of-thought reasoning does not lower it, even though it sharply strengthens the model;
  • the lower bound of the data itself is unknown: zero cannot be reached, and as the score nears the floor, progress on the scale will slow, even though each small fraction will mean a bigger jump in intelligence.

What this means in practice

For developers and product managers, the takeaway is this: benchmarks saturate quickly, and most older tests were "solved" at roughly the GPT-4 level. The pretraining metric remains a more stable reference point, and OpenAI proposes making it a common language for the industry. When a lab announces a new base model, it makes sense to ask not only about benchmark scores but also about the prediction error on an open dataset.

Whether the metric becomes a standard depends on competitors. Publishing internal figures gives them a clear picture of OpenAI's position, and the company knowingly took on this risk in pursuit of shared safety standards.

Story

Key Questions Answered

What perplexity is, in a simple example?

Suppose a model sees the start of the code line def add(a, b): return a and must predict the continuation. If it almost always produces + b, its prediction error is low. If it guesses at random, the error is high. Perplexity averages this error across a huge set of texts the model did not see during training. In practice, OpenAI looks at the log of perplexity, also known as cross-entropy. These are not accuracy percentages: the lower the number, the better the model predicts.

Why limits, and what comes next?

The author names the next window as roughly 0.68–0.70. He openly admits he does not know what abilities will emerge there or whether we will even be able to test for them. He also expects that the usual alignment audits will work worse, and that models will get better at recognizing that they are being tested. Therefore, OpenAI proposes an industry standard.

What are the key details on Limitations of the metric?

Pachocki himself lists the drawbacks: Labs use different validation sets and data mixtures, so direct comparison is not yet possible; OpenAI calculates perplexity on a sample from its internal code, and for GPT-2 and GPT-3 such a measurement poorly reflects general capabilities; perplexity does not correspond to any particular applied task, and chain-of-thought reasoning does not lower it, even though it sharply strengthens the model; the lower bound of the data itself is unknown: zero cannot be…

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…