Multilingual Transcripts and Distillation Help AI Better Detect Emotion in Speech

Researchers proposed a multimodal method for analyzing speech sentiment that combines audio and automatically generated multilingual transcripts through cross-modal transformers, as well as a way to transfer this knowledge to an audio-only model using knowledge distillation. The paper was published on arXiv on July 7, 2026.

Why Determining Speech Sentiment is Difficult

Automatic recognition of positive or negative sentiment in speech requires simultaneous analysis of voice intonations and interpretation of spoken words. Existing solutions typically rely only on audio foundation models, and it remains unclear whether they account for both aspects simultaneously.

How the Proposed Method Works

The authors combined audio and text information through cross-modal transformers. Text transcripts are automatically created by a speech recognition tool (ASR), then translated into multiple languages using machine translation — resulting in multiple text modalities.

• Audio and multilingual text features are combined by a cascading architecture of cross-modal transformer blocks that integrate modalities one by one
• Knowledge from the multimodal model ("teacher") is transferred through distillation to a single-modality audio model ("student")
• Experiments on a large-scale dataset showed that automatically generated text information provides significant accuracy gains in sentiment classification
• Ablation studies confirmed that both automatic transcripts and automatic translations are useful
• The audio model can be improved through distillation without additional computational overhead at inference time

Code for reproducing the results is available as open source on GitHub.

What This Means

The method shows that multilingual text transcripts, generated automatically, are useful for analyzing emotions in speech even when the final model works only with audio. This means such systems can be deployed where a transcript is unavailable during inference, for example in real-time scenarios, while maintaining quality "inherited" from the heavier multimodal model.

Frequently Asked Questions

Why are translations of the transcript into other languages needed if

the model understands the language of the recording anyway? According to the authors' ablation studies, automatic translations of transcripts into multiple languages, along with the transcripts themselves, provide an additional boost to sentiment classification accuracy — both sources proved useful.

Does the model need text during actual use?

No — thanks to knowledge distillation from the multimodal "teacher" to the audio model "student," the final model determines speech sentiment from audio alone, without transcripts and without additional computational overhead.