Multilingual transcripts and distillation help AI detect sentiment in speech more accurately

A team of researchers has proposed a multimodal method for speech sentiment analysis. It fuses audio with automatically generated multilingual transcripts through cross-modal transformers, then uses distillation to transfer that knowledge to an audio model that needs no transcripts. The paper was published on arXiv on July 7, 2026.

Why speech sentiment is hard to detect

Automatically recognizing sentiment in speech, whether positive or negative, requires analyzing vocal intonation and interpreting the spoken words at the same time. Existing solutions typically rely on audio foundation models alone, and it remains unclear whether they capture both aspects at once.

How the proposed method works

The authors combined audio and text information using cross-modal transformers. An automatic speech recognition (ASR) tool generates the text transcripts, and machine translation then renders them into several languages, producing multiple text modalities.

• A cascaded architecture of cross-modal transformer blocks fuses audio and multilingual text features, integrating the modalities one after another
• Distillation transfers knowledge from the multimodal model (the "teacher") to a unimodal audio model (the "student")
• Experiments on a large-scale dataset showed that automatically generated text information significantly improves sentiment classification accuracy
• An ablation study confirmed that both the automatic transcripts and the automatic translations help
• Distillation can improve the audio model without adding computational overhead at inference time

The code to reproduce the results is publicly available on GitHub.

What it means

The method shows that automatically generated multilingual transcripts help with emotion analysis in speech even when the final model works with audio only. That means such systems can be deployed where transcripts aren't available at inference time, such as in real-time settings, while retaining quality "inherited" from a heavier multimodal model.

FAQ

Why translate the transcript into other languages if the model already

understands the recording's language? According to the authors' ablation study, automatic translations of the transcripts into several languages provide an additional gain in sentiment classification accuracy on par with the transcripts themselves; both sources proved useful.

Does the model need text during real-world use?

No. Thanks to knowledge distillation from the multimodal "teacher" to the audio "student" model, the final model detects speech sentiment from audio alone, without transcripts and without additional computational overhead.