Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

AI reviewer at EMNLP wrote reviews three times as long as human ones

A lengthy AI review can easily be mistaken for a thorough assessment. In an overview of EMNLP, AIRI researcher Maria Maryina describes an experiment in which a GPT-5.4-based system produced reviews with a median length more than three times that of human reviews. According to the papers' authors, the models summarized the content well but offered less convincing criticism. For teams, this is a reason to check which comments actually affect the decision.

AI-processed from Habr AI; edited by Hamidun News
AI reviewer at EMNLP wrote reviews three times as long as human ones
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1AI reviews in the experiment did not affect whether papers were accepted.
Key metric #2Authors' assessments do not replace independent verification of the criticism.
Key metric #3A model's comments should be ranked by their impact on the main conclusion.

You give a model a product hypothesis and receive twenty comments: add a comparison, expand the sample, test three more scenarios. It looks like thorough work. But which of these requests could change the launch decision? In an overview of EMNLP, AIRI researcher Maria Maryina describes a similar problem with automated reviews of research papers. The case is useful for anyone who uses AI to assess other people's work.

According to her account, one system produced reviews with a median length more than three times that of human reviews. Yet the papers' authors found the models' criticism less convincing. The result is a familiar situation: there is more text, but a person still has to work out the priorities.

What the organizers tested

The experiment described involved three AI reviewer configurations: the AAAI system based on GPT-5.4, and ReviewerToo using GPT-OSS-120B and GLM-4.7 Flash. Authors opted in to receive generated reviews and assess their quality. The committee deciding which papers to accept did not see these reviews. The experiment therefore examined the usefulness of feedback, rather than the ability to select research papers automatically.

According to the assessments cited in the overview, the models did a good job of conveying the research content and identifying strengths. They performed less well on critical comments. The GPT-5.4-based system was particularly verbose; the gap in length compared with human reviews was smaller for the other two configurations.

Maryina also recounts complaints from authors who declined to participate: AI fixates on details and demands additional experiments while undervaluing the novelty of an idea. It is important here to distinguish the sources of these conclusions. These are comments from people who opted out, rather than a measured result from a comparison of all reviews. The overview itself also provides no basis for treating every request made by a model as mistaken.

Why a list of comments can get in the way

Imagine a team testing whether a new search tool helps employees find instructions faster. The model suggests increasing the sample size, comparing several ranking methods, adding other languages and measuring results after six months. Every recommendation sounds reasonable. But they carry different weight for the immediate decision.

If the initial measurement of search time is unreliable, the main conclusion is indeed in doubt. Testing another language may be needed later, when the team enters a new market. Put both comments on a single mandatory checklist, and a small pilot can easily turn into a project lasting months. This illustrates a practical risk; it is not a finding of the EMNLP experiment.

The same logic applies to peer review: a missing key comparison can invalidate a conclusion, while an additional test may simply broaden its applicability. In the overview, Maryina suggests paying closer attention to a study's contribution and distinguishing necessary checks from requests to make an already substantial piece of work even larger.

However, a long review proves nothing on its own. It may contain useful findings. An author's dissatisfaction does not mean the reviewer is wrong either: an unwelcome comment sometimes points to a real weakness. Every substantive objection needs to be checked against the material in the paper.

How to apply this in a team

The experience described suggests a practical approach: ask the model to connect every comment to a specific decision. Which conclusion does it challenge? Which passage in the document supports it? What is the minimum check needed to resolve the issue? What would change if the comment proved valid?

This makes it easier for a person to identify errors that block a decision and improvements that can wait. For example, a request for one more experiment should explain which uncertainty it would resolve. Otherwise, the team risks spending time on persuasively worded wishes.

To assess the AI reviewer itself, it is useful to examine how many of its comments proved valid and how many significant errors it missed. This is an editorial recommendation for practical use; the overview cited does not report these metrics. It shows something narrower: participants rated the summaries and descriptions of strengths more highly than the quality of the criticism.

It therefore already makes sense to entrust a model with an initial assessment, provided the team checks its results. But deciding whether there is enough evidence for the next step requires an understanding of the task, the cost of further verification and the consequences of an error. The length of a generated review does not automatically reduce that work.

Story

Key Questions Answered

What the organizers tested?

The experiment described involved three AI reviewer configurations: the AAAI system based on GPT-5.4, and ReviewerToo using GPT-OSS-120B and GLM-4.7 Flash. Authors opted in to receive generated reviews and assess their quality. The committee deciding which papers to accept did not see these reviews. The experiment therefore examined the usefulness of feedback, rather than the ability to select research papers automatically.

Why a list of comments can get in the way?

Imagine a team testing whether a new search tool helps employees find instructions faster. The model suggests increasing the sample size, comparing several ranking methods, adding other languages and measuring results after six months. Every recommendation sounds reasonable. But they carry different weight for the immediate decision. If the initial measurement of search time is unreliable, the main conclusion is indeed in doubt. Testing another language may be needed later, when the team enters a new market.

How to apply this in a team?

The experience described suggests a practical approach: ask the model to connect every comment to a specific decision. Which conclusion does it challenge? Which passage in the document supports it? What is the minimum check needed to resolve the issue? What would change if the comment proved valid? This makes it easier for a person to identify errors that block a decision and improvements that can wait. For example, a request for one more experiment should explain which uncertainty it would resolve.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…