AI agents re-checked 2,000+ ICML 2026 papers and found errors that reviewers missed

Peer review at top ML conferences relies heavily on trust. A reviewer gets several papers and a couple of weeks, and usually won't run someone else's code just to verify one table. In summer 2026, this was put to the test head-on, with AI agents brought in to help.

What happened

Hugging Face, together with alphaXiv, a service for discussing scientific publications, ran the ICML 2026 Open Reproductions hackathon. The task: take the papers accepted to the conference (there were more than two thousand) and try to reproduce the reported results. Participants did this using AI agents. They found errors in the papers that the reviewers had not noticed.

ICML is one of the main machine learning conferences, and an accepted paper is often seen as a mark of quality. But a reviewer sees a PDF, and the attached code often goes unrun: there is neither the time nor the incentive.

It is important to note what the original announcement does not say. It does not state how many papers were fully reproduced, how many errors were found, or how serious they are: a typo in a formula, or a result that falls apart. Without these figures, it is too early to say that the agents "refuted" some share of science. But the format itself is telling: re-checking two thousand papers used to be blocked by a shortage of people and time.

How it works in practice

The workflow for an agent is roughly this: read the paper, find the repository or reconstruct the method from its description, run the code, get the numbers, and compare them with the table in the text. Each step on its own is tedious, but these are exactly the steps where results break.

Suppose a paper claims a 3-percentage-point gain on a benchmark. The agent runs the code with the stated settings and gets a gain of half a point. From there, several scenarios are possible: a hyperparameter in the text doesn't match the code, test data leaked into training, or the baseline model was tuned worse than the authors' own. A human who doesn't run the code won't see this. An agent doesn't get tired and can run hundreds of checks in parallel.

These are typical causes of discrepancies, not a list of the hackathon's findings: the specific types of errors are not disclosed in the source material.

What developers and product managers should do about it

Don't treat a table in a paper as a guarantee. If you are building a product on a "state of the art" result from a fresh paper, its numbers remain a hypothesis. Budget time for your own reproduction, and first check whether someone has already tried it.

Apply the same approach inside your company. An agent that takes your internal experiment report or model evaluation and tries to repeat it from scratch based on the description is a cheap way to find the gap between what was written and what was actually run. For teams that publish benchmarks and comparisons, it is almost a ready-made QA stage.

Watch the market. For founders, the hackathon highlights a niche: reproducibility-checking tools, methodology checkers, and research audits for investors and corporate R&D departments.

Limitations

An agent that failed to reproduce a result has not yet proven the authors wrong. The cause may be a lack of compute, closed data, an incomplete description of the method, or a mistake by the agent itself. That is why findings need to be confirmed by a human. A realistic setup is one where the agent gathers candidate errors and an expert delivers the verdict.

There is also the opposite risk. A flood of false positives would bury authors in complaints that have to be sorted out by hand, and trust in automated checks would quickly fall.

Reviewers are still here, but now they have a partner who actually runs the code. Reproducibility is ceasing to be a luxury and becoming an automatable procedure, something like CI for science. Teams that depend on fresh research should get used to this before the authors of the papers do.