Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

CogAlign teaches Qwen3-VL-4B to pick the odd one out like humans: 53% → 57% agreement

A neural network recognizes a mousetrap, a matchbox and an aardvark, but it picks the "odd one out" differently than a human does. When answering directly, Qwen3-VL agreed with people in 37–41% of cases, against a random baseline of 33%. The CogAlign method trains the model on the features people use to make that choice (66 semantic dimensions derived from 4.7 million responses). It raised Qwen3-VL-4B's agreement from 53.37% to 56.81%. Putting the same list of features in the prompt did not help: the model needs worked examples with explanations.

AI-processed from Habr AI; edited by Hamidun News
CogAlign teaches Qwen3-VL-4B to pick the odd one out like humans: 53% → 57% agreement
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1A list of 66 features in the prompt gave 52.70%, against 53.37% without it
Key metric #2Centaur, with 70 billion parameters, scored 51.96%, below all three Qwen3-VL models
Key metric #3On MMBench, the 4B model's accuracy fell from 82.96% to 81.39%

Show a person a mousetrap, a matchbox and an aardvark, and within seconds they will call the aardvark the odd one out: humans made two of the objects, while the third is alive. A multimodal neural network, meaning a model that works with both images and text, recognizes all three objects but may pick the odd one out on an entirely different basis. In the researchers' test, Qwen3-VL models answering directly matched the human choice in only 37–41% of cases, while random guessing in a three-option task gives 33.

3%. The method in question, CogAlign, fine-tunes a model on the features people use to make their choice. As a result, Qwen3-VL-4B's agreement with humans rose from 53.

37% (with standard chain-of-thought reasoning) to 56.81%. For developers, this is a warning sign: if a product groups items, moderates images or searches for similar content, a model that "sees" the world differently from the user will make mistakes where a human finds the answer obvious.

Why standard alignment does not solve the problem

Models are usually aligned with RLHF, where people rate responses and a separate reward model is trained on those ratings, or with DPO, where the model learns directly from pairs of accepted and rejected responses. This works well for stopping a bot from being rude and for teaching it a tidy format. But a final answer that matches the human one says nothing about how the model got there.

The model might have noticed that two objects are made of metal. Or it might have simply guessed, because similar images appeared more often in a certain context in the training data. The authors of the paper Aligning MLLMs with the Latent Structure of Human Cognition propose teaching the model not only the answer but also the basis for the choice.

Where the 66 features came from

The foundation is the THINGS project: 1854 concepts of real-world objects and a large odd-one-out experiment. People were shown three images and asked to pick the one least similar to the other two. This produced about 4.7 million human judgments. The SPoSE algorithm placed the objects in a multidimensional space and extracted 66 dimensions a human can interpret: food, animals, textiles, electronics, transport, plants, round objects, fluffy and soft things, weapons and danger, medicine. A mousetrap scores high on the "metallic or artificial" dimension, an aardvark on the "animal" dimension. These are not 66 switches in the brain but a mathematical model built from people's responses.

How the training works

CogAlign works in three stages. First, for each triplet of images, the Jackknife method removes one dimension at a time and checks how the probability changes that the remaining pair is judged the most similar. The dimension with the strongest effect becomes the key feature, for example "human-made".

Next, a standard LLM turns this feature into a coherent explanation; it does not look for the correct answer but is handed a ready-made criterion. Finally, the multimodal model is fine-tuned on the resulting examples (SFT). The training set contains 309 329 examples: about 80 thousand THINGS cognitive tasks, and the rest are image recognition, storytelling, difference spotting and other tasks, so that the model does not become a narrow specialist in finding the odd one out.

Results and pitfalls

The evaluation used 453 642 image triplets. For Qwen3-VL-4B, agreement was 41.48% with a direct answer, 53.37% with standard chain-of-thought reasoning and 56.81% after CogAlign. For the 8B model, the figures are 40.46%, 52.28% and 56.99%. For 30B-A3B, which uses a Mixture of Experts architecture, they are 37.09%, 55.38% and 56.21%. Centaur, a specialized 70-billion-parameter model trained on human behavioral data, scored 51.96%, and all three Qwen3-VL models outperformed it after CogAlign. On a sample of about 75.6 thousand tasks, Qwen2.5-VL-7B improved from 56.63% to 59.29%, and Gemma3-12B from 55.77% to 57.28%.

The most practical finding concerns prompts. If all 66 dimensions are inserted into the system prompt, Qwen3-VL-4B scores 52.70%, against 53.37% with standard chain-of-thought reasoning. When the model reasons through a single selected dimension, accuracy drops to 49.01%. The models often got stuck in loops: they listed that the aardvark is not metallic, not artificial, not flat and not green until they ran out of tokens. Each statement is true, but almost none of them bears on the task. Replacing the full explanations with short dimension labels drops the result to 42.81%.

There is no universal gain. On MMMU, results improved for all three Qwen3-VL versions, but on MMBench the 4B model's accuracy fell from 82.96% to 81.39%. In addition, the 66 dimensions include stereotypically feminine and stereotypically masculine features, so this data should be checked before use.

The takeaway for teams tuning multimodal models: if you want a model to choose "like a human", a long list of rules in the prompt may not help and can even hurt. It is more reliable to fine-tune on examples where each answer comes with an explanation, and to mix them with ordinary tasks to preserve general skills.

Story

Key Questions Answered

Why standard alignment does not solve the problem?

Models are usually aligned with RLHF, where people rate responses and a separate reward model is trained on those ratings, or with DPO, where the model learns directly from pairs of accepted and rejected responses. This works well for stopping a bot from being rude and for teaching it a tidy format. But a final answer that matches the human one says nothing about how the model got there. The model might have noticed that two objects are made of metal.

Where the 66 features came from?

The foundation is the THINGS project: 1854 concepts of real-world objects and a large odd-one-out experiment. People were shown three images and asked to pick the one least similar to the other two. This produced about 4.7 million human judgments. The SPoSE algorithm placed the objects in a multidimensional space and extracted 66 dimensions a human can interpret: food, animals, textiles, electronics, transport, plants, round objects, fluffy and soft things, weapons and danger, medicine.

How the training works?

CogAlign works in three stages. First, for each triplet of images, the Jackknife method removes one dimension at a time and checks how the probability changes that the remaining pair is judged the most similar. The dimension with the strongest effect becomes the key feature, for example "human-made". Next, a standard LLM turns this feature into a coherent explanation; it does not look for the correct answer but is handed a ready-made criterion.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…