ARC-AGI

ARC-AGI (Abstraction and Reasoning Corpus for AGI) is a benchmark of visual analogy puzzles requiring systems to infer transformation rules from a few input-output grid examples and apply them to new grids — tasks humans solve easily but that proved highly resistant to AI approaches for years.

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) is a benchmark designed by François Chollet and presented in his 2019 paper "On the Measure of Intelligence." Each task presents a small number of input-output pairs of colored grids (up to 30×30 cells, 10 colors) that share a hidden transformation rule; the system must identify the rule and apply it to a new input grid to produce the correct output. Rules involve abstract concepts such as symmetry, object counting, spatial translation, pattern completion, and boundary detection. No predefined vocabulary of operations is provided — the rule must be induced entirely from the examples, making memorization of specific patterns ineffective.

The benchmark is explicitly designed so that virtually any adult human can solve the tasks with minimal instruction (human scores are typically 80–85%) while requiring fluid reasoning — the ability to form new abstractions from very few examples — rather than crystallized knowledge that can be acquired through training data exposure. This distinction is Chollet's central theoretical point: ARC-AGI measures sample-efficient generalization that he argues is closer to general intelligence than performance on knowledge-retrieval benchmarks. The public dataset comprises 400 training tasks and 400 evaluation tasks.

ARC-AGI gained broad attention in 2024 when Chollet and entrepreneur Mike Knoop launched the ARC Prize, a $1M competition. The contest highlighted how poorly standard approaches performed — most systems, including large language models with chain-of-thought prompting, scored under 30% before the competition. OpenAI's o3 model, using high-compute test-time search over candidate programs, achieved approximately 87.5% on the semi-private ARC-AGI evaluation in late 2024, widely reported as a landmark result and the subject of substantial debate about its implications for general intelligence.

In early 2025, the ARC Prize Foundation released ARC-AGI-2, a substantially harder successor specifically designed to resist the test-time search strategies that enabled o3's result. As of 2026, frontier models score below 10% on ARC-AGI-2's public tasks, maintaining its role as a highly challenging open problem. The benchmark series has become a central reference in discussions of fluid reasoning, the nature of AI generalization, and the distance between current systems and human-level general intelligence.