KAIST's AI Reads Lips and Gets Only 19.4% of Words Wrong
Lip reading is hard: "p", "b" and "m" look almost identical, and conventional models write a sentence left to right and cannot correct an early mistake. KAIST's model first recognizes unambiguous words like "BAGS" and "TRIP" and uses them to tell "PACK" from "BACK." It gets 19.4% of words wrong, about one in five, the best result among methods trained only on LRS3. When the start of a text is ambiguous, the model generates the confident parts first.
AI-processed from TechXplore AI; edited by Hamidun News
Imagine watching a video with the sound off: a person moves their lips, and you try to work out whether they said "pack" or "back." For a camera and a neural network, these are nearly indistinguishable. Researchers at KAIST (the Korea Advanced Institute of Science and Technology) have developed a way to handle this: the model first recognizes the words it is confident about, then uses them to guess the uncertain ones.
As a result, the system gets 19.4% of words wrong. This is the share of words that are misrecognized, omitted, or inserted compared with the reference transcript.
Put simply, about one word in five. According to the authors, this is the best score among methods that use only the LRS3 dataset for labeled training.
Why lip reading is so easy to get wrong
The core problem is that different sounds look the same. In English, /p/, /b/ and /m/ are nearly indistinguishable without audio: "pack" and "back" look virtually identical on video. Humans cope by using context. If the sentence at the end is about bags and a trip, a person understands that "pack" was meant, not "back."
Conventional models can't do this. They generate text one word at a time, left to right, like phone autocomplete. If the first word is chosen wrongly, the error carries forward, and the model cannot go back and fix it. The beginning of the sentence, the most ambiguous part, becomes the weak link for the whole structure.
How KAIST's approach works
The team led by Professor Yong Man Ro used a diffusion language model (DLLM). It works differently: the sentence is first represented as a set of empty, masked positions, and the model fills them in over several passes. The filling order does not have to run left to right.
The authors exploited this. First, the model identifies the words it is sure of, meaning the positions where lip movements are unambiguous. In the press release example, these are "BAGS" and "TRIP." It then uses them as anchors and, drawing on context from both sides, decides what stood in the uncertain position: "PACK" rather than "BACK." With each step, fewer positions remain undetermined.
Training was split into two stages. First, the model learns to map lip movements in video to text. Then it separately learns to estimate sentence length. During recognition, it uses the video's duration to propose several plausible lengths, builds a candidate sentence for each, and selects the best one. The selection rests on two measures: how probable that length is and how confident the model is in its own words.
Results and limits
The model was trained on LRS3, which contains 433 hours of video with exact transcripts. The resulting 19.4% word error rate is, according to the authors, the lowest among methods that use only LRS3 for labeling. The caveat matters: the comparison is within the same modest amount of labeled data. Systems trained on much larger datasets are not part of this ranking.
There are other limitations. This is research published as an arXiv preprint, not a finished product. The article says nothing about processing speed, model size, or performance under poor filming conditions (side angles, bad lighting, a closed mouth). The authors themselves acknowledge that the model is still heavy. Their next step is to make it smaller and cheaper to run, so it can operate in real time on wearable devices and robots without an internet connection.
What to take away in practice
There are two practical use cases. The first is transcribing speech where noise drowns it out, for example on industrial sites with loud machinery. The second is assistive devices for deaf and hard-of-hearing people.
For those building systems on language models, the broader lesson is more useful: the order of generation can also be configured. If a task has ambiguous spots that later context clarifies, it's worth trying diffusion models or schemes that generate the confident parts first and the doubtful parts after. Ro himself argues that flexible control over generation order alone removes a fundamental constraint tied to visual ambiguity. To verify this claim in production, independent tests on more diverse data and latency measurements on real hardware are needed.
- Nov 21, 2026MLPerf Training Adds Nvidia Rubin and TPU Ironwood in First Appearance: 153 Systems
- Nov 21, 2026Waymo pulls robotaxis off highways after new entries into work zones
- Nov 21, 2026Google Opens Pre-Orders for Gemini Smart Glasses from $395, Shipping December 10
- Nov 21, 2026Space Data Centers Are Still Unprofitable: Launch Costs Run 3–19 Times Over the Break-Even Point
Key Questions Answered
Why lip reading is so easy to get wrong?
The core problem is that different sounds look the same. In English, /p/, /b/ and /m/ are nearly indistinguishable without audio: "pack" and "back" look virtually identical on video. Humans cope by using context. If the sentence at the end is about bags and a trip, a person understands that "pack" was meant, not "back." Conventional models can't do this. They generate text one word at a time, left to right, like phone autocomplete.
How KAIST's approach works?
The team led by Professor Yong Man Ro used a diffusion language model (DLLM). It works differently: the sentence is first represented as a set of empty, masked positions, and the model fills them in over several passes. The filling order does not have to run left to right. The authors exploited this. First, the model identifies the words it is sure of, meaning the positions where lip movements are unambiguous.
What are the key details on Results and limits?
The model was trained on LRS3, which contains 433 hours of video with exact transcripts. The resulting 19.4% word error rate is, according to the authors, the lowest among methods that use only LRS3 for labeling. The caveat matters: the comparison is within the same modest amount of labeled data. Systems trained on much larger datasets are not part of this ranking. There are other limitations. This is research published as an arXiv preprint, not a finished product.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.