Stroop Test Shows VLMs Read the Text in an Image Before They See the Color

A group of researchers has published the preprint arXiv 2610.12650 on how vision-language models choose between two equally valid features of the same image. They took the Stroop test, in which the word "RED" is printed in blue and the subject is asked to name the color. The direction of the result was the same for contrastive and generative VLMs: when the two conflict, the models more often answer with the written word rather than the color. The authors also checked the cause. In embedding space, text is encoded more strongly and is more separable than color.

How the experiment works

A standard VLM benchmark asks what is in the picture. This task is trickier: the image contains two features, both relevant to the question, but pointing in different directions. These are Stroop conflict stimuli: the word names one color but is set in another. If a model faithfully follows the instruction "name the color," the word should not affect its answer.

The authors translated the paradigm into a benchmark format and ran several models from two families through it:

• contrastive CLIP-style models, in which the image and text are matched in a shared space;
• generative VLMs, which answer the question in free-form text;
• the conflict condition: the text and the color do not match, and the task requires reporting the color;
• the behavioral metric: the share of answers based on the text versus the share based on the color.

The abstract gives no numerical values, so the size of the bias will have to be checked in the full paper. The qualitative conclusion is stated unambiguously: the bias toward text is stable across models, meaning it does not depend on a particular architecture.

Where the bias comes from

The most interesting part starts after the behavioral results. The authors "opened the black box" and compared how text and color features are represented in the embeddings. Text signals turned out to be expressed more strongly and to be more separable. Color in the same space is weaker and blurrier.

This asymmetry works as a predictor. When the researchers tried to steer the model with an intervention in representation space, it was substantially easier to push it toward text directions than toward color directions. In other words, the difference shows up not only in the answers but also in how readily the model yields to intervention.

One possible explanation is that the models are trained on huge collections of image-caption pairs, where text in images often carries meaning. But that is a hypothesis, not a conclusion of the paper: the authors show where the bias lives in the representations, but do not prove which data it grew out of. The main takeaway is stated cautiously: different features within a single image have different prominence, or saliency, for the model.

What to check in your pipeline

For products where a VLM reads documents, screenshots, packaging, road signs, or interfaces, this is a practical risk. If an image contains text and the question concerns a visual property of an object, the model may answer from the text. The classic example is a label naming one color on an object of a different color.

Developers should do a few things:

• add conflict cases to the test set, where the text in the image contradicts the visual feature;
• not treat a prompt like "answer by color" as a guarantee, since the bias sits in the representations;
• for sensitive tasks, such as quality control or safety, split OCR and visual analysis into separate steps;
• factor this in when assessing resistance to text-injection attacks through images: judging by the results, text in a picture influences the model more than you might expect.

The study does not offer a ready-made cure and does not claim the problem will disappear with scaling. It does provide a simple, reproducible diagnostic test borrowed from cognitive psychology. You can run it on any new VLM before it reaches production.