Researchers propose a new way to measure AI intelligence beyond the human level

In July 2026, a group of researchers published a paper on arXiv proposing a new way to measure the capabilities of AI systems that already outperform humans. Instead of fixed tests written by people, models generate public challenges for one another, and the results are aggregated into an adversarial psychometric rating.

What is wrong with existing benchmarks

The question in the paper's title, how to measure intelligence beyond the human scale, is becoming more practical as flagship models overtake people one by one on professional exams, programming tests and science olympiads. Human-written benchmarks saturate over time: models gradually reach the maximum or near-maximum score, and the test stops distinguishing systems by quality. For models that already surpass humans in a specific area, the problem gets worse: test designers (examiners) may simply not know which tasks are both hard enough and verifiable for a system at that level.

The authors argue that this difficulty is not an accident of particular tests but a systemic consequence of the approach itself, absolute-scale evaluation, in which a result is compared against a fixed human reference.

How the new approach works

Instead of an absolute scale, the authors propose a paradigm of relative measurement: models generate public challenges designed to separate other systems by quality. In other words, the test is created not by a human expert but by an AI system that competes with the others.

• The results of these challenges are aggregated into an adversarial psychometric rating system
• By the authors' design, the system can scale as the capabilities of the evaluated systems grow, so it does not hit a ceiling the way static benchmarks do
• The authors describe practical protocols that reduce incentives for private-information attacks, meaning attempts to peek at answers or test conditions
• The protocols support judge-free adjudication, meaning they require neither a separate arbiter model nor a human to deliver a verdict
• The framework has been tried both on verifiable tasks and on open-ended, non-verifiable domains

Why such an evaluation system is needed

The idea is that evaluation should be able to keep working after systems pass the human frontier in a given domain. There, human-written benchmarks can no longer separate a good model from an excellent one, because the human test designer cannot reliably judge the difficulty and correctness of a task for a superhuman system.

The key engineering challenge of this approach is to prevent models from colluding with one another or from unfairly tailoring challenges to competitors' weak spots. That is why the authors separately describe protocols against private-information attacks and insist on adjudication without a separate judge: the fewer points in the system that coordinated action by participants can "fool", the more reliable the final rating.

What this means

The work fits into the broader debate over saturation of existing AI benchmarks. As models overtake human tests one after another, the industry needs measurement methods that do not depend on humans' ability to devise sufficiently hard tasks. Having models generate and evaluate challenges for each other is one possible answer, although its practical reliability and resistance to manipulation have yet to be tested on real, widely used systems.