smol-audio from Deep-unlearning: A collection of Colab notebooks for audio model fine-tuning

Deep-unlearning has released smol-audio — an open collection of Jupyter notebooks for practical work with modern audio models directly in Google Colab. The project is assembled as a set of reproducible recipes for those who need not to read abstract reviews, but quickly fine-tune ASR, run audio captioning, and analyze multimodal pipelines.

How smol-audio is Organized

The main idea of smol-audio is simple: instead of yet another generalizing framework, the team created a flat repository of independent notebooks, where each solves one specific task. All scenarios are built on the Hugging Face stack — transformers, datasets, peft, and accelerate — and are tailored to run without local GPU setup. Open Colab, connect the runtime, and get a working starting point, not a collection of scattered snippets from issue trackers. This makes the project more like an engineering cookbook than a showcase demo.

An important detail is transparency. In smol-audio, they don't hide the training loop and data preparation behind convenient wrappers, so engineers can see how batches are organized, where loss is calculated, and what exactly changes during fine-tuning. For newcomers, this is educational material; for experienced teams, it's a convenient base for adaptation to their own dataset.

According to the authors, most recipes fit within the 16 GB of Colab memory, meaning they don't require expensive infrastructure from the first step.

What Models Are Inside

Currently, the collection primarily covers fine-tuning ASR models, but is not limited to speech recognition alone. The repository and accompanying review include scenarios for several architectures that differ significantly in structure and training requirements. This is precisely useful: instead of a universal "do it somehow," the user gets working templates for a specific class of model. This format lowers the entry barrier when you need to quickly test a hypothesis on your own audio dataset.

• Whisper — adaptation to a new language or narrow domain.
• Parakeet from NVIDIA — fine-tuning a CTC model, including a variant with LoRA.
• Voxtral from Mistral — ASR tuning with prompt masking for an LLM architecture.
• Granite Speech from IBM — an example of language fine-tuning on the Italian YODAS-Granary corpus.
• Audio Flamingo 3 and PE-AV — audio captioning tasks, zero-shot video classification, and audio-to-text retrieval.

The difference between these models is not cosmetic. Whisper works as a sequence-to-sequence system and generates transcription token by token. Parakeet relies on a CTC approach, which is usually easier and faster for inference, but requires different logic for aligning audio frames with text. Voxtral is closer to speech understanding based on LLM, so prompt masking is critical there: loss should be calculated on the transcription, not on the text prompts themselves.

The review also separately notes a scenario for Dia-1.6B, tailored for dialogue TTS.

Why Engineers Need This

The strongest side of smol-audio is not a list of big names, but savings on routine engineering. When a team takes on a new audio model, weeks often go not to research but to basic things: properly assembling a dataset, not confusing preprocessing, choosing the right fine-tuning mode, and not hitting GPU memory limits. Here the authors immediately show both full fine-tuning and a lighter variant through LoRA, which is especially important for large audio and multimodal models.

This is noticeable in more complex scenarios as well. For Audio Flamingo 3, the project shows how to fine-tune a model for audio description tasks, useful for accessibility, content indexing, and media library search. For Meta PE-AV, multimodal inference is demonstrated with a shared embedding space for audio, video, and text: this approach allows zero-shot video classification and cross-modal search without separate task-specific fine-tuning.

In other words, smol-audio is useful not only for ASR but for a broader voice AI and multimodal ecosystem.

What This Means

smol-audio transforms work with audio AI from a set of scattered experiments into an understandable practical collection of recipes. If the trend toward voice assistants, multimodal models, and local adaptation to languages persists, such repositories will become basic infrastructure for ML teams: not a replacement for research, but a short path from idea to the first working prototype.