Whisper and Gemma 3 Linked With Contrastive Learning for Cheap Voice Input in LLMs

Adding voice input to an LLM cheaply turned out to be less simple than many multimodality papers suggest. The author of the experiment tried to link the Whisper audio encoder and the Gemma 3 language model through a compact projector, and after a series of failures arrived at a working configuration based on contrastive learning.

How the stack was built

The idea was simple: rather than train an expensive multimodal system from scratch, take a ready-made audio encoder and a ready-made LLM and connect them with a "translator" between embedding spaces. The encoder was Whisper Medium, because its internal representations are better suited to speech recognition than those of self-supervised alternatives. On the text side the author used Gemma 3 4B, and the bridge was a two-layer MLP projector that compressed audio vectors and translated them into the LLM's embedding space.

To avoid training the model only on clean studio English, the training stream was assembled from several datasets and mixed dynamically. This exposed the system to speech of varying quality, language and delivery from the start. The article stresses that the mix is there not for nicer statistics, but so that from the first epochs the system does not get used to a single acoustic environment and a single language. Otherwise any deviation, whether noise, a pause or a Russian fragment, would immediately break recognition.

• LibriSpeech train.360 as the core of the corpus
• LibriSpeech train.100 as additional clean English
• Russian LibriSpeech for Russian speech
• DisfluencySpeech with pauses, slips and stutters

Why everything kept breaking

The first bet was the most obvious recipe: teacher forcing and plain cross-entropy over the transcript. The LLM received an instruction, the audio vectors and the correct text, and the loss was computed only on the answer tokens. In practice the setup barely heard the recording: the model produced incoherent fragments, and WER could get stuck around 300%. Even after adding LoRA it was clear the problem ran deeper: the projector was not delivering the audio signal to where the language model knows how to read it. Gemma kept too strong a prior on the familiar geometry of text tokens.

A series of targeted fixes followed. The author added a stage zero in which Gemma first simply learned to copy text on instruction, since the model used was not the instruction-tuned version. Then came experiments with quantization and regularizers: a commitment loss was meant to keep projector outputs near known embeddings, SWD to pull the distributions of audio and text vectors together, an entropy loss to push the system to use more codes, and VICReg to keep individual dimensions from collapsing.

t-SNE visualizations helped reveal two main ailments: representation collapse and a geometric gap between the audio and text spaces. But each new tweak cured only one symptom. SWD improved the shape of the distribution without improving its content. The entropy loss broadened code usage, but did so randomly. VICReg raised variance, yet the vectors scattered chaotically. Again and again the system found a workaround in which the metrics looked locally better while no real recognition appeared. That became the main lesson of the regularization phase: with a weak primary signal, the model optimizes the math, not the meaning.

What finally worked

The turning point was abandoning the idea that alignment could be coaxed out through indirect penalties alone. The author made contrastive learning the main signal and switched to symmetric InfoNCE: an audio vector must be closer to its own transcript than to every other text in the batch, and vice versa. Unlike the earlier regularizers, this loss specifies not general statistics but concrete pairwise relations.

With a large batch it worked noticeably better: the loss curve fell smoothly with no sharp jumps, and WER dropped to 35%. The result does not yet reach the level of commercial ASR systems, but it is no longer random noise. In the logs the model began to make phonetically plausible errors: it picked up how words sound and stumbled more like a person with poor hearing than like a broken text generator. For a first pass this matters more than the absolute WER figure: the system stopped imitating an answer and started actually using the sound. The author sees this as the main sign of progress.

"But the main thing is that it already hears."

What this means

This case shows that cheap audio modality for local LLMs is possible, but not through the "magic" MLP projector from the papers. A simple pairing of a ready-made encoder and an LLM starts to work only when a strong alignment signal appears between them. For developers this is an important takeaway: if you want to add voice to your own model without costly training from scratch, a contrastive stage may be not an option but a mandatory foundation.