Fish Audio Drama 3 brings whispers and voice breaks to local TTS within a single paragraph
Small local TTS models with 0,5–1,7B parameters read a dramatic monologue as cheerfully as API documentation: they smooth over whispers and voice breaks. The author tried Praat and audiobook references before running Fish Audio Drama 3 (35B-A3B MoE) on a Mac. The model sighs on its own when prompted with [sighs], drops into a whisper, and builds intensity through the final questions.
AI-processed from Habr AI; edited by Hamidun News
Local TTS models now comfortably handle straightforward narration. Feed in a clean voice sample, mark the word stress, and you get a decent voiceover for a lecture or video tutorial. The problems start when the task calls for acting: within a single paragraph, the voice needs to move through irony, a lump in the throat, a whisper, and a break. Models with 0,5–1,7B parameters stick to the reference's average intonation and read tragedy in the same tone as documentation.
The author, a teacher, was building an interactive course and wanted to move beyond monotonous narration. The test was Hamlet's monologue with Yorick's skull: four sentences that require four changes in delivery. First comes warm nostalgia, then a constricted voice and disgust, then a hushed whisper, and finally a series of increasingly forceful questions.
What didn't work
The starting point was a Russian-language fine-tune of Qwen3-TTS 1.7B with word-stress marks. The speech was clean and intelligible but emotionally flat. Its loudness plot is almost a horizontal line across the entire text.
The second attempt split the monologue into phrases, generated each with its own reference, and adjusted the pitch in Praat through parselmouth using the TD-PSOLA algorithm. This exposed two problems. First, the model copies not just the reference's intonation but also the distance from the microphone and the room acoustics, so the joins make it audible that the lines were recorded in different rooms. Second, whispers and vocal constriction are not simply matters of pitch: the vocal folds stop closing, the clean fundamental tone disappears, and breath noise emerges. Simply shifting F0 produces mechanical modulation, not a whisper. Praat remains useful for small corrections, such as word stress or the intonation of a question.
The third hypothesis was to use professional narrators from the ToneBooks dataset as references. The result was a solid audiobook, but not drama. A narrator protects their vocal folds and maintains a steady delivery for hours; they do not act out vocal constriction or disrupted breathing.
How Drama 3 differs from lightweight TTS models
Fish Audio released the fishaudio/drama-3 weights in late October 2026, and a 4-bit port arrived in mlx-audio in early November. Under the hood is a 35B-parameter MoE model with 256 experts per layer, of which 8 are active per token (about 3 billion active parameters). It also has an audio tower with its own temporal memory and a 44,1 kHz codec.
The key practical finding concerns references. The template accepts several separate text-and-audio pairs: calm speech, whispering, a heavy exhalation. Concatenate everything into a single file, and a conventional zero-shot model averages the states or jumps erratically between tones. Drama 3 ties each state to its own transcript.
The second technique concerns annotation. In ref_text, the [sighs] and <whisper>...</whisper> tags must appear exactly where the actor sighs or whispers in the recording, and the words must match the speech verbatim; otherwise, alignment breaks down. A tag in the main text then triggers the specific breath and whisper the model heard from that person.
Results and cost
Across three voice profiles, the model performed every transition. The [sighs] tag produces an actual inhalation with a change in pace and pitch, rather than just a pause. Inside <whisper>, it produces a hushed whisper with audible airflow. In the final questions, Drama 3 builds a rising intonation pattern on its own until the voice breaks. At temperature=0.8, the recording length varies from 40,5 to 44,0 s, but there were no hallucinations or dropped words.
Here is the cost. The mixed-quantization weights occupy 22,15 GiB, and peak memory use on an M4 Pro with 48 GB reaches 28,6–28,8 GB. Loading takes 8,3 s, and generation runs at roughly 1:1 real time. On a graphics card with 24 GB of VRAM, a model like this hits the memory ceiling.
The smaller S2 Pro (4B, 8-bit, 6,73 GB on disk) initializes in 2,4 s and peaks at 12,45 GB, using 2,3 times less memory. It can whisper too: like Drama 3, its level drops to the bottom of the plot. But it accepts one reference array and single-word tags such as [sigh], [sad], and [whisper], so the three references have to be concatenated with a 0,25 s pause.
To give all the tracks a consistent target timbre, the author ran them through RVC v2 (40 kHz). The settings protect=0.35 and rms_mix_rate=0.25 prevent the converter from trying to turn the whisper into voiced, “sung” speech. The downside is that some high frequencies are cut compared with the native 44,1 kHz audio. The author considers this a reasonable tradeoff: RVC can be trained on 10–15 minutes of recordings, while fine-tuning a 35B model is incomparably harder.
What follows from this
If you need expressive voice acting rather than narration, look for models with a multi-reference template and performance tags instead of trying to fix flat intonation in post-processing. Prepare references of the same person in different emotional states and annotate them with the same tags you use in the main text. If you have less than 30 GB of memory, S2 Pro is worth trying first. The limitation of this article is that the tests used a single monologue on a Mac, and the author did not test Drama 3's multi-speaker mode.
- Nov 12, 2026Google opens preorders for Gemini glasses starting at $395 and blocks covert recording
- Nov 12, 2026Samsung Is Preparing a Paid Galaxy AI Subscription: From $4.99 a Month and Credits Instead of Unlimited Use
- Nov 12, 2026Adobe showcases four AI prototypes at Max: 3D from photos, a sound square, and copy-and-paste edits
- Nov 12, 2026Instagram launches prompt-based filter creation for users aged 18+ in the US
Key Questions Answered
What didn't work?
The starting point was a Russian-language fine-tune of Qwen3-TTS 1.7B with word-stress marks. The speech was clean and intelligible but emotionally flat. Its loudness plot is almost a horizontal line across the entire text. The second attempt split the monologue into phrases, generated each with its own reference, and adjusted the pitch in Praat through parselmouth using the TD-PSOLA algorithm. This exposed two problems.
How Drama 3 differs from lightweight TTS models?
Fish Audio released the fishaudio/drama-3 weights in late October 2026, and a 4-bit port arrived in mlx-audio in early November. Under the hood is a 35B-parameter MoE model with 256 experts per layer, of which 8 are active per token (about 3 billion active parameters). It also has an audio tower with its own temporal memory and a 44,1 kHz codec. The key practical finding concerns references. The template accepts several separate text-and-audio pairs: calm speech, whispering, a heavy exhalation.
What are the key details on Results and cost?
Across three voice profiles, the model performed every transition. The [sighs] tag produces an actual inhalation with a change in pace and pitch, rather than just a pause. Inside <whisper>, it produces a hushed whisper with audible airflow. In the final questions, Drama 3 builds a rising intonation pattern on its own until the voice breaks. At temperature=0.8, the recording length varies from 40,5 to 44,0 s, but there were no hallucinations or dropped words. Here is the cost.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.