OpenAI found critical errors in the popular SWE-Bench Pro test

OpenAI conducted an analysis of SWE-Bench Pro — a popular benchmark suite for evaluating AI models' programming skills — and discovered critical issues: approximately 30% of tasks contain errors. As a result, the company withdrew its support and recommendation for this benchmark.

What is SWE-Bench Pro

SWE-Bench Pro (Software Engineering Benchmark Pro) is a standardized test suite developed to objectively measure the ability of AI models to solve real programming tasks. The test is widely used by researchers, companies, and independent evaluators as one of the primary tools for comparing the performance of various coding models.

Tasks in SWE-Bench Pro are based on real challenges from GitHub and require the model not merely to write code, but to understand complex software systems, fix bugs, and work with non-trivial scenarios. It is precisely because of this closeness to reality and objectivity of evaluation that the test gained wide popularity in the AI community and became the de facto standard for public comparison of models.

30% of tasks contain errors or defects

The analysis conducted by OpenAI revealed serious problems in approximately 30% of SWE-Bench Pro tasks. These errors — from incorrect test cases to improperly formulated tasks and insufficiently correct reference solutions — make the test results unreliable and misleading when assessing the real capabilities of models.

The discovery of such large-scale problems in a popular and authoritative benchmark is a significant blow to the reliability of this assessment tool. If 30% of all tasks is roughly 90 or more erroneous tests, this is an enormous amount of compromised data. OpenAI, which previously approved and publicly recommended SWE-Bench Pro as a reliable test for comparing models, now officially withdraws its support. This means that results obtained based on this benchmark and widely cited in scientific papers, blogs, and comparative reviews should be interpreted with great caution.

Why this is fundamentally important

Reliable and unbiased tests are the foundation of fair assessment of AI models' capabilities. When a benchmark contains systematic errors, results become incomparable, and companies or researchers may draw incorrect conclusions about which model is truly better.

This incident underscores that even popular and widely used tests require constant review, validation, and updating. Companies and researchers who relied on SWE-Bench Pro results when selecting models or evaluating their systems should reconsider their conclusions and be more cautious in interpreting this data.

• Approximately 30% of tasks in SWE-Bench Pro contain errors or defects
• OpenAI withdrew its official support and recommendation of the benchmark
• The test was widely used by researchers and companies for comparing coding models

What this means

The identified problems in SWE-Bench Pro demonstrate a fundamental challenge in assessing AI capabilities: creating truly representative, unbiased, and error-free tests is exceptionally difficult and requires enormous effort. The community should work closely to develop more reliable evaluation methods that reflect the true capabilities of models without artifacts and errors in the tests themselves.

This may lead to a revision and re-evaluation of comparative results of various AI models that were previously published based on SWE-Bench Pro. Significant additional efforts will be required to develop and independently verify improved benchmarks for evaluating the programming skills of AI systems.