Tencent releases Youtu-Parsing-Omni: one model parses documents, audio and video

Tencent has released Youtu-Parsing-Omni, an "all-purpose parser": the model reads document pages, photos, audio and video and converts them into a structured record. In that record, every fragment carries coordinates on an image or a time interval. The record serves as a perception layer for AI agents. The agent converts the input to text once, and from then on any language model, including one with no vision or hearing, works with that text and cites a specific region or interval as evidence. Code and weights are openly available.

Why agents need pre-parsed input

LLM-based agents increasingly rely on data outside their own memory: they search for sources, call tools and operate computers through the screen. This data rarely arrives as clean text. More often it is PDFs, screenshots, call recordings and videos.

Feeding raw pixels and audio to the agent's main model creates two problems:

• each time the model returns to the material, it re-encodes long sequences of visual and audio tokens; • an answer derived from pixels or a waveform does not itself show which segment supports it.

Parsing-Omni parses the input once. The agent then works with the text record and can check each claim against its anchor.

Existing solutions cover this task only partially. Document parsers handle ordinary pages well, but they usually leave chemical structures and sheet music to narrow, specialized models. Multimodal models are tuned for describing images and answering questions, not for structured output with grounding. For different input types, you have to maintain different models with different response formats.

One encoder instead of two towers

In existing omni parsers, including Logics-Parsing-Omni, two separately trained encoders encode pixels and audio. Their streams meet for the first time inside the language model. That causes three problems: a frame and the sound from the same instant are not matched during perception, the two encoders keep time independently, and each new modality requires a new tower with its own pretraining.

Youtu-Omni-Encoder is designed differently:

• one shared transformer with bidirectional attention for all modalities, initialized with the weights of a pretrained text language model to read text in images better; • each modality has only lightweight "input legs" and output projectors. Images and frames are cut into patches with a single linear transform, while audio is converted into a log-mel spectrogram and a few compressing convolutions; • the decoder is a dense text model trained with the Youtu-LLM-2B recipe. It outputs text only.

In this design, a new modality needs a new "leg" and projector, not a second tower. The shared weights still have to be fine-tuned on the new input.

How frames and audio are aligned in time

The encoder uses a unified position system. The frequency range of rotary position embeddings (RoPE) is divided into segments for width, height and time. A frame and an audio token receive a position on a common timeline.

A shared timeline is not enough; the modalities also need to interact. So some layers work as fusion layers. A video's frames and audio fragments are packed in chronological order, and a new attention window starts at every "audio → frame" boundary. A frame sees the audio from the same moment. This way, "which words are spoken while this slide is on screen" is computed at the perception stage, and the decoder does not have to reconstruct it after the fact.

In the remaining layers, each frame and audio fragment has its own window. Both modes use the same weights. During training, video passes through fusion with a set probability, so both routes work at inference.

One schema and self-training on its own outputs

The common output format is called OmniSchema. It is JSON with three required fields: a list of localized units with content, a global description of the input, and service fields. For images, a unit is an element with a bounding box in normalized coordinates; for audio and video, it is a segment with a start and end (accurate to the second) and a speaker identifier that is consistent across the whole input. The schema covers 10 tasks and three input types: images (pages and ordinary photos), audio and video.

The unified format made it possible to apply a single fine-tuning procedure, OSAD (OmniSchema-Aware On-Policy Distillation). After standard training on reference answers and training with a verifiable reward, the model learns from its own parses. The "teacher" is a frozen copy of the model itself, which is additionally shown the reference answer. It provides target distributions for each token. Tokens are split by role, either content or structure, and each role gets its own loss function. After each round, the trained model becomes the new teacher. No hand-built reward function or separate teacher model is needed.

Results

A single checkpoint serves all 10 tasks. The authors claim the following:

• the best result on OmniDocBench v1.6, a benchmark for parsing document pages; • first place among open models on OmniParsingBench, which tests parsing of images, audio and video. The model outperformed Logics-Parsing-Omni, with the biggest gap on audio and video; • the best overall score on chart parsing; • the highest share of exact matches of chemical molecular "fingerprints" (the [email protected] metric) among the compared systems, including DeepSeek-OCR 2; • in character-level transcription of printed sheet music, the model matches the specialized LEGATO and far outperforms general-purpose systems.

For engineers, the takeaway is this: instead of a stack of OCR, ASR and video models with different formats, you can run one parser with verifiable source references. For business, it is a path to agents that process contracts, meeting recordings and videos and show where each claim came from. All figures are the authors' claims; there are no independent measurements yet.