VLM trained robots to read human emotions — but trust matters more than politeness

Scientists from the University of Melbourne taught collaborative robots to read human emotions using a vision-language model (VLM). Research published in IEEE Robotics and Automation Letters shows: robots become more attentive to people, but trust in them is still determined not by manner of communication, but by their ability to handle tasks.

Context Instead of Facial Expressions

Standard human-machine interaction systems rely on facial expression analysis and object tracking in frames. Researchers led by Sung Chan Hong decided to go further and use VLM — a class of models that process both text and visual data simultaneously.

To train the model, volunteers watched video recordings where robots handed objects to people with varying degrees of success and described the emotions of the scene participants. Crucially, the raters took the entire context into account: a furrowed brow in a person tapping their fingers on the table means irritation; the same furrowed brow in a person focused on a task simply means concentration.

Comparison with a classical algorithm yielded convincing results: VLM achieved an accuracy of 0.86 on a scale from 0 to 1, whereas the traditional approach scored 0.77. According to Hong, the model "saw the entire scene: where the person is located, what they are doing, how they interact with the robot" — this is precisely what gave it the advantage.

Personalized Apologies Work — But Not Always

In the second experiment, 40 volunteers worked collaboratively with a robot that was pre-programmed to make an error. After the failure, the robot responded in one of two ways:

• Adaptive apology — formulated based on the robot's reading of the person's emotional state
• Standard phrase — a pre-written scripted response

31 out of 40 participants preferred the personalized response. This confirms: people care about feeling that the robot notices their state and responds to it meaningfully.

However, survey data revealed another side of the picture: the majority of participants' trust ratings for the robot decreased after the error regardless of the type of apology.

"Personalized apology works like a social lubricant, but it does not restore trust lost due to failure in a physical task,"

Hong explains.

Where VLM's Capabilities End

Analysis of the second experiment's data revealed an important limitation. When VLM's emotion ratings were compared to what participants themselves reported about their emotional state, the model's accuracy dropped sharply. The model aligned well with the perceptions of outside observers, but poorly predicted the internal experiences of the participants themselves.

"VLM is a good observer of external social signals, but cannot read minds," Hong explained. In other words, the model notices what an outside observer looking from the side would notice. In situations where a person masks their emotions or experiences something not reflected in their facial expressions and gestures, the system fails.

What This Means

The research outlines a clear priority for developers: first reliability and accuracy in task execution, then the layer of emotional interaction. People are willing to collaborate with robots that can apologize in a human-like manner, — but first and foremost they want competent partners who do not make mistakes.

As VLM-based approaches develop, the gap with traditional emotion recognition systems will grow, but this does not resolve the fundamental question of trust.