Scientists proposed new way to measure AI intelligence above human level

A group of researchers in July 2026 published a work on arXiv proposing a new approach to measuring the capabilities of AI systems that already surpass humans: instead of fixed tests written by humans, the models themselves generate public trials for each other, and the results are aggregated into a competitive psychometric rating.

What's the Problem With Existing Benchmarks

The question posed in the work's title — "how to measure intelligence beyond human scale" — becomes increasingly practical as flagship models one by one outpace humans on professional exams, programming tests, and scientific olympiads. Benchmarks written by humans become saturated over time (saturate) — models gradually accumulate maximum or near-maximum scores on them, and the test ceases to distinguish systems by quality. For models that already surpass humans in a particular area, the problem worsens: test creators (examiners) may simply not know which tasks are simultaneously sufficiently difficult and verifiable (verifiable) for a system of that level.

The authors argue that this difficulty — not a chance occurrence of specific tests, but a systemic consequence of the approach to evaluation on an absolute scale (absolute-scale evaluation), when the result is compared against a fixed human standard.

How the New Approach Works

Instead of an absolute scale, the authors propose a relative measurement paradigm (relative measurement): the models themselves generate public trials (public challenges) designed to separate other systems by quality — that is, the test is created not by a human expert, but by the AI system itself competing with others.

• The results of such trials are aggregated into a competitive psychometric rating system (adversarial psychometric rating system)
• Such a system, according to the authors' design, is capable of scaling along with the growth of the capabilities of the evaluated systems — that is, it does not "hit the ceiling" like static benchmarks
• The authors describe practical protocols that reduce incentives for attacks based on private information (private-information attacks) — that is, attempts to "peek" at answers or test conditions
• The protocols support adjudication without the participation of a judge (judge-free), that is, they do not require a separate model-arbiter or human to render a verdict
• The framework is tested on both verifiable tasks and open, not subject to formal verification (open-ended non-verifiable) domains

Why Such an Evaluation System Is Needed

The idea is that evaluation could "continue to work" even after systems surpass the human level (human frontier) in a particular area — where benchmarks written by humans are already unable to distinguish a good model from an excellent one, because the test writer themselves cannot guarantee to evaluate the complexity and correctness of a task for a superhuman system.

The key engineering challenge of such an approach is to prevent models from "colluding" with each other or tailoring tests to their weaknesses of competitors in unfair ways. This is precisely why the authors separately describe protocols against attacks based on private information and insist on adjudication without a separate judge: the fewer points in the system that can be "tricked" by coordinated actions of evaluation participants, the more reliable the resulting rating.

What This Means

The work fits into a broader discussion about the saturation of existing AI benchmarks: as models overtake human tests one by one, the industry needs measurement methods that do not depend on humans' ability to come up with sufficiently complex tasks. An approach where models themselves generate and evaluate trials for each other is one possible answer to this challenge, although its practical reliability and resistance to manipulation still need to be tested on real, widely used systems.

Frequently Asked Questions

What is the core idea behind the new benchmark approach?

Instead of using fixed human-written tests, the models themselves generate public challenges for each other and are ranked in a competitive psychometric rating system. This allows continuous measurement even as models surpass human performance levels.

How does this approach handle the saturation problem?

By having models generate tests dynamically based on competing with other systems, the approach inherently scales with the capability of the systems being evaluated. As models become stronger, more challenging tests are naturally generated, preventing the "ceiling" problem of static benchmarks.

What safeguards prevent models from gaming the system?

The authors describe protocols against private-information attacks and support judge-free adjudication, minimizing manipulation points in the rating system and ensuring the reliability of the final ranking.