Synthesized Voice Read a Novel in 1 h 58 min; the Surrounding Pipeline Code Was What Broke

A synthesized voice read an entire novel: 16 chapters, 15 444 words, 1 h 58 min of audio. According to the author of a Habr article series, the voice stayed steady for the full two hours. Almost all the breakages turned up not in the model itself but in the code around it. The 14-phrase test set used to compare model versions caught none of them.

The text for the experiment was the novel «Merdiven» («Staircase») by Shamil Alyadin. This is the third article in the series: the first contained a table of results from reading the book, and in the second the author promised a separate breakdown of what broke over the long distance. He does not disclose model names, data sources, internal scripts, or training and decoding parameters. What the text does offer is the metrics, the testing method, and the mistakes he ran into.

Where things actually broke

The author lists five problem areas. All of them belong to the pipeline code, meaning what happens before and after audio generation:

• splitting text into sentences;
• stitching short phrases together;
• trimming breaths;
• extracting text from the book file;
• timing markup.

Engineers who have worked with TTS will recognize the list. The model receives a clean chunk of text and produces a clean chunk of audio. If the text is cut in the wrong place, a short line («— No.») is glued to the wrong neighbor, or a breath is clipped mid-syllable, the listener hears the defect even though the model's own metrics have not changed. A text extraction error happens before synthesis even starts: the model faithfully voices whatever it is given.

Why 14 phrases noticed nothing

The author's second conclusion is arguably more interesting than the first. The set of 14 test phrases was used to compare model versions and, by all appearances, did that job. But it could not see long-distance problems. That is understandable: a single phrase has no joins between sentences, does not require splitting a chapter, and does not test how timing markup accumulates over two hours of audio.

There is also a more troubling case. In one episode, the set could not tell a good fix from a harmful one. For a team that makes decisions based on such tests, this is an alarming signal: a metric that cannot distinguish improvement from regression is just noise.

The numbers help gauge the scale. 15 444 words over 118 minutes works out to about 131 words per minute, and the average chapter runs roughly 965 words and 7 minutes of audio. Over that distance there are far more joins between fragments than in any short test set, and every join is a potential point of failure.

What to check in your own pipeline

The piece suggests some practical takeaways, and they do not depend on any particular model.

• Keep at least one long run alongside the short test set: a chapter or a whole book. A short set works for comparing model versions, but not for acceptance testing of the entire pipeline.
• Test the pipeline code separately from the model: segmentation, stitching, pause trimming, text extraction, and timing markup. Each stage needs its own checks.
• Start defect analysis with the text that goes into synthesis. If the error is there, blaming the model is pointless.
• Check the test itself: if it cannot tell a useful fix from a harmful one, it is too early to make decisions based on its results.

The author promises a more detailed account of the testing method in the article itself, and that will be the main value of the piece. For now, the announcement points to one simple conclusion: when a voice sounds stable for two hours straight, the quality of the pipeline comes down to how carefully the seams are built.