VLMs teach robots to read human emotions, but trust matters more than politeness

Researchers at the University of Melbourne have trained collaborative robots to read human emotions using a vision-language model (VLM). The study, published in IEEE Robotics and Automation Letters, shows that robots are becoming more attentive to people, but trust in them is still determined not by how they communicate but by whether they can handle their tasks.

Context instead of facial expressions

Standard human-machine interaction systems rely on facial expression analysis and object tracking in the frame. The researchers, led by Seung Chan Hong, decided to go further and use a VLM, a class of model that processes text and visual data at the same time.

To train the model, volunteers watched videos in which robots handed objects to people with varying degrees of success, and described the emotions of the people in the scene. Crucially, the raters took the whole context into account: a furrowed brow on a person drumming their fingers on a table signals irritation, while the same furrowed brow on someone focused on a task simply signals concentration.

The comparison with the classical algorithm produced a convincing result: the VLM reached an accuracy of 0.86 on a scale from 0 to 1, while the traditional approach scored 0.77. According to Hong, the model "saw the whole scene: where the person is, what they are doing, how they interact with the robot," and that was what gave it the advantage.

Personalized apologies work, but not always

In the second experiment, 40 volunteers worked together with a robot that had been programmed in advance to make a mistake. After the failure, the robot responded in one of two ways:

• Adaptive apology: composed based on the person's detected emotional state
• Standard phrase: a pre-written scripted response

31 of the 40 participants preferred the personalized response. This confirms that people want to feel the robot notices their state and responds to it in a meaningful way.

However, the survey data revealed another side of the picture: trust ratings for the robot dropped for most participants after the error, regardless of the type of apology.

"A personalized apology works as a social lubricant, but it does not restore trust lost through failure at a physical task,"

Hong explains.

Where the VLM's capabilities end

Analysis of the second experiment's data exposed an important limitation. When the VLM's emotion ratings were compared with what participants themselves reported about their state, the model's accuracy fell sharply. The model agreed well with the perceptions of outside observers but predicted participants' inner experiences poorly.

"A VLM is a good observer of external social signals, but it cannot read minds," Hong said. In other words, the model notices the same things an outsider watching from the side would notice. In situations where a person masks their emotions or experiences something not reflected in facial expressions and gestures, the system fails.

What this means

The study sets a clear priority for developers: reliability and accuracy in task execution first, the emotional interaction layer second. People are willing to work with robots that can apologize in a human way, but above all they want competent partners that do not make mistakes.

As VLM-based approaches develop, the gap with traditional emotion recognition systems will likely widen, but it will not remove the fundamental question of trust.