Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents (Nov 23, 18:30 MSK): 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseInsight · On RadarTechCrunch→ original

OpenAI's Chief Scientist Proposes an AI Speed Limit: GPT-4 Perplexity at 0.91, Now Below 0.8

To limit how fast AI develops, the industry needs a shared speedometer. Jakub Pachocki of OpenAI proposes perplexity, which measures how accurately a model predicts the next token (a piece of a word). The lower the number, the stronger the model. GPT-4 scored about 0.91, and models are now below 0.8, a level at which, by his account, strong cybersecurity and math capabilities emerged. Companies could agree not to cross that threshold until they learn to control their models.

AI-processed from TechCrunch; edited by Hamidun News
OpenAI's Chief Scientist Proposes an AI Speed Limit: GPT-4 Perplexity at 0.91, Now Below 0.8
Source: TechCrunch. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTechCrunch ↗
Editor-in-Chief
Key metric #1Altman called the post "the most important thing OpenAI has published in a while"
Key metric #2Applied skill tests go stale quickly, while perplexity remains useful
Key metric #3Anthropic CEO Dario Amodei had earlier published the idea of a speed limit

Imagine a speed limit on a road, but cars have no speedometers: each driver judges "fast" in their own way. That is roughly the situation the conversation about slowing AI development has found itself in. OpenAI Chief Scientist Jakub Pachocki has proposed such a speedometer: perplexity. The metric shows how accurately a language model predicts the next token (a piece of a word) in ordinary text. The lower it is, the better the model. GPT-4 scored about 0.91, and OpenAI's models have now dropped below 0.8. Below, we look at why this particular metric matters and what follows from it.

Why a "speedometer" is needed at all

The topic of slowing AI has stopped being the domain of activists alone. A former OpenAI and Anthropic researcher said that companies are "playing with our lives." Anthropic's IPO filing states plainly that the technology may carry "catastrophic or existential risks for society." Anthropic CEO Dario Amodei published a plan to "set the pace of the frontier," which, judging by the reaction, was backed by Sam Altman and Elon Musk. Amodei proposed that companies agree on shared safety standards and limits on the speed of uncontrolled progress.

But to set a speed limit, you first have to agree on how to measure it. That is the question Pachocki's post addresses. Altman called it "the most important thing OpenAI has published in a while."

Why perplexity instead of familiar benchmarks

Tests of applied skills (solving problems, writing code, olympiad math) go stale quickly: models hit a ceiling, and the test stops distinguishing a new version from the previous one. Perplexity, according to Pachocki, remains "informative" while other benchmarks come and go. Put simply, it measures how rarely a model is "surprised" by the actual continuation of a sentence. It has no connection to the startup Perplexity, which simply shares the name.

The post includes a chart showing how perplexity fell across OpenAI's models starting with GPT-3.5. Pachocki points to two markers. GPT-4, at around 0.91, could already reason productively through chain-of-thought. That prompted the team to deliberately develop this skill and build modern reasoning models. Now, with the prediction loss (called "loss" in the post) below 0.8, he says, "transformative" capabilities have appeared in cybersecurity and mathematics.

From there comes the idea: perplexity could become the basis of a "common intelligence benchmark," and on that basis companies would set "shared limits." They would pledge not to cross a certain threshold until the ability to understand and monitor AI behavior reaches a specific mark. The pause is tied not to the calendar but to how well people have learned to see what is happening inside models.

What remains off the table

For now, this is one company's proposal, not an industry agreement. The account from TechCrunch names no specific threshold, no mechanism for verifying that a competitor is not circumventing the limit, and no answer to who would calculate the numbers.

The practical questions are obvious. Perplexity is computed on a specific set of texts and with a specific tokenizer, so an honest comparison of models from different companies requires a shared test corpus. Closed labs would have to share at least this measurement. Moreover, the metric relates to real skills only indirectly. The cybersecurity and mathematics example shows a correlation, not a guarantee: which exact capabilities will appear at the next drop is unknown in advance.

What this means for developers and product teams

For now, the takeaway is mostly strategic. If the idea becomes an industry standard, the pace of releasing new frontier models could depend on a publicly agreed threshold, not only on hardware readiness and budget. That would affect the roadmaps of API providers and how quickly teams can plan around the next generation of models. Three things are worth watching: whether OpenAI names a specific number, whether Anthropic and Google join the scheme, and whether an independent body emerges to verify those figures.

Story

Key Questions Answered

Why a "speedometer" is needed at all?

The topic of slowing AI has stopped being the domain of activists alone. A former OpenAI and Anthropic researcher said that companies are "playing with our lives." Anthropic's IPO filing states plainly that the technology may carry "catastrophic or existential risks for society." Anthropic CEO Dario Amodei published a plan to "set the pace of the frontier," which, judging by the reaction, was backed by Sam Altman and Elon Musk.

Why perplexity instead of familiar benchmarks?

Tests of applied skills (solving problems, writing code, olympiad math) go stale quickly: models hit a ceiling, and the test stops distinguishing a new version from the previous one. Perplexity, according to Pachocki, remains "informative" while other benchmarks come and go. Put simply, it measures how rarely a model is "surprised" by the actual continuation of a sentence. It has no connection to the startup Perplexity, which simply shares the name.

What remains off the table?

For now, this is one company's proposal, not an industry agreement. The account from TechCrunch names no specific threshold, no mechanism for verifying that a competitor is not circumventing the limit, and no answer to who would calculate the numbers. The practical questions are obvious. Perplexity is computed on a specific set of texts and with a specific tokenizer, so an honest comparison of models from different companies requires a shared test corpus. Closed labs would have to share at least this measurement.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…