Kimi releases PerceptionBench: none of 16 MLLMs reaches 60%

The Kimi team at Moonshot AI has released PerceptionBench, a benchmark of 3000 verified questions that measures one thing: what a multimodal model actually sees in an image. Among sixteen frontier MLLMs, none reached 60% accuracy. On average, models struggle most with perception-related hallucinations. The dataset and evaluation code are open on GitHub.

A taxonomy built from failures, not theory

Benchmark authors usually define a set of skills first and then write questions to fit it. The PerceptionBench team went the other way. They collected errors made by frontier models on 42 existing benchmarks and, for each one, found the earliest faulty step. The error came first, the category second. That process produced ten atomic abilities.

The team then selected the most telling failures and broke them down into simple sub-questions. Some questions were written from scratch on additional images. The internal pool grew to 17000+ verified questions. The release includes 3000 of them: 60% come from decomposing real model errors, and 40% were newly written. The categories are balanced by size, and difficulty is stratified.

The main requirement is that the answer can be found by looking, with no reasoning or outside knowledge. Examples from the announcement: which hour mark on a clock face the Gemini symbol sits on, how many potted plants are in the frame (reflections in the glass don't count), which of eight fragments fills the cutout in a pattern, and what appeared in the second photo of the same scene.

The authors also note that the source benchmarks overlap only weakly. The average pairwise weighted Jaccard is 0.20. Each one catches a narrow slice of perception errors, and none covers the whole picture.

Ten categories and their weights

The 3000 questions in the release are distributed as follows:

• Depth & 3D — 330 (11.00%)
• Visual Counting — 330 (11.00%)
• Visual Relation — 330 (11.00%)
• Visual Attribute — 330 (11.00%)
• Visual Localization — 330 (11.00%)
• Fine-grained Recognition — 290 (9.67%)
• Visual Comparison — 279 (9.30%)
• Hallucination — 271 (9.03%)
• Context Integration — 255 (8.50%)
• OCR — 255 (8.50%)

The tests target specific weaknesses. Models are asked to read the largest white text while preserving case and punctuation, or to find text from normalized coordinates. They count cubes in a stack, including hidden ones, and compare line thickness on a chart. The Hallucination category works differently: in questions about white desks in an office or dog logos among 15 patterns, the correct answer is zero. A model that tends to "fill in" objects gets these wrong.

What the results show and how to use them

The announcement summarizes the results as follows. None of the sixteen participants reached 60%. Perception hallucinations turned out to be the weakest ability on average. Models with nearly identical overall scores differ noticeably in what exactly they can distinguish. This matters when choosing a model: the overall score hides the profile, and the profile determines where a model will break on your task.

The available text of the announcement contains no per-model results table, so you will need to check the repository for specific rankings. The benchmark was published by Moonshot itself, which is a player in this market. It should therefore be re-verified on your own data rather than accepted as an independent arbiter.

The practical takeaway for teams embedding MLLMs in products is simple. If you are building document workflows, look at OCR and Localization. For robotics and navigation, Depth & 3D and Visual Relation matter more. For retail and inventory, Counting is critical. For anything where an error costs money, you need the Hallucination category: it shows how prone a model is to seeing things that aren't there. The evaluation code is open, so you can run your candidates across the categories and compare their profiles in an evening. That is more reliable than comparing a single total on a leaderboard.