Multilingual Transcripts and Distillation Help AI Detect Emotion in Speech More Accurately

A group of researchers has proposed a multimodal method for speech sentiment analysis. It combines audio with automatically generated multilingual transcripts through cross-modal transformers, and uses distillation to transfer that knowledge into an audio model that needs no transcripts. The paper was published on arXiv on July 7, 2026.

Why speech sentiment is hard to detect

Automatically recognizing sentiment in speech, whether positive or negative, requires analyzing vocal intonation and interpreting the spoken words at the same time. Existing solutions usually rely only on audio models (audio foundation models), and it remains unclear whether they take both aspects into account simultaneously.

How the proposed method works

The authors combined audio and text information through cross-modal transformers. Text transcripts are generated automatically by a speech recognition (ASR) tool and then translated into several languages by machine translation, producing multiple text modalities.

• Audio and multilingual text features are merged by a cascaded architecture of cross-modal transformer blocks that integrate the modalities one after another
• Knowledge from the multimodal model (the "teacher") is transferred by distillation into a single-modality audio model (the "student")
• Experiments on a large-scale dataset showed that automatically generated text information gives a significant gain in sentiment classification accuracy
• An ablation study confirmed that both the automatic transcripts and the automatic translations are useful
• The audio model can be improved through distillation without extra computational load at inference time

The code to reproduce the results is publicly available on GitHub.

What this means

The method shows that automatically generated multilingual text transcripts are useful for emotion analysis in speech, even when the final model works with audio alone. This means such systems can be used where a transcript is unavailable at inference time, for example in real time, while keeping the quality "inherited" from the heavier multimodal model.

Frequently asked questions

Why translate the transcript into other languages if the model already

understands the language of the recording? According to the authors' ablation study, automatic translations of transcripts into several languages, alongside the transcripts themselves, give an additional gain in sentiment classification accuracy. Both sources proved useful.

Does the model need text during actual use?

No. Thanks to distillation from the multimodal "teacher" to the audio "student" model, the final model determines speech sentiment from audio alone, without transcripts and without extra computational load.