Apple ML Research Studies Alignment in Multimodal LLMs and Their Tendency to Hallucinate

Apple ML Research has published a comprehensive study of preference alignment in multimodal language models (MLLMs), systems that process text and images at the same time. The authors argue that alignment, which has become standard practice for text-only LLMs, remains far less studied in the multimodal setting.

Why MLLM hallucinations are harder than those in ordinary LLMs

Multimodal models hallucinate differently from text-only LLMs. According to Apple ML Research, hallucinations in image-understanding systems take two forms: factual errors, where the model states incorrect facts about the world, and content inconsistency, where the answer sounds convincing but contradicts what is actually shown in the picture.

The second type is especially insidious. If a model describes a blue car in a photo as red, or calls a summer shot a winter one, that is not a gap in knowledge but a failure of "vision." Alignment for multimodal systems is aimed precisely at this problem.

"The main goal of alignment for MLLMs is to encourage these models to align their responses with image information," the

Apple ML Research publication states.

How preference alignment works

Preference alignment is a fine-tuning technique in which a model is shown pairs of answers and annotators mark which one is "preferable" according to set criteria. In text-only LLMs, this approach, including RLHF (Reinforcement Learning from Human Feedback) and its derivatives, became the de facto standard after InstructGPT and ChatGPT. For multimodal systems, almost no comparable systematic research has been done to date.

Key points of the work:

• Preference alignment is critically important for LLM quality, but its effect in MLLMs has barely been studied
• In MLLMs, alignment solves an additional task: tying the model's answer to the actual visual content
• Hallucinations in MLLMs fall into two types: factual and visual (inconsistency with the image)
• Apple ML Research conducts a systematic comparative analysis of alignment techniques as applied to image-understanding tasks

According to the study, this work is one of the first comprehensive attempts to close the gap between how well alignment is understood for text models and how little is known about its effect on multimodal systems.

Why this matters for the industry

Multimodal models are now used in medical diagnostics, content moderation and consumer assistants, from Google Lens to Apple Intelligence. In each of these scenarios, a hallucination of the "seeing what isn't there" kind costs far more than a text error: a misread X-ray or a misidentified object on the road has direct consequences.

The Apple ML Research study arrives as multimodal systems move from lab prototypes to mass deployment. Understanding how alignment shapes the accuracy of a model's "vision" is becoming a fundamental question for the whole industry.

What this means

The Apple ML Research work points to a systemic gap: the industry is actively building and deploying multimodal models, yet the mechanism for aligning them with the real content of images has remained understudied. If the study's findings form the basis of new training methods, they will directly affect the quality of products that work with visual content, including Apple Intelligence and similar systems.