Peking University speeds up video loading for AI by 1.45x

Researchers at Peking University led by academician Wen Gao have proposed codec4AI: a way of storing and reading video tailored to the actual workload of neural network training. On a large robotics corpus, the DDRA H.265 variant showed 1.45x the data-loading throughput compared to the LeRobot v3 configuration the authors used as a baseline. Media volume, meanwhile, was 58.7% smaller, with comparable reconstruction quality. The paper is published on arXiv (2610.33079), and the code and documentation are openly available.

Why codecs get in the way of training

Standard video codecs were built for playback: frames run in sequence, and compression squeezes maximum efficiency out of references between neighbors. Most frames are stored as a difference relative to other frames, so showing just one requires reconstructing the chain of references back to the nearest keyframe.

Training works differently. Robotics datasets for VLA models and other multimodal systems sample frames randomly across time. The dataset loader requests a frame from the middle of a clip, and the decoder dutifully unpacks everything needed to produce it.

The authors call this effect decode amplification and identify two sources:

• sequential reconstruction of unrelated frames that no one asked for;
• reconstruction of the reference chains needed for the target frames themselves.

Common workarounds hit storage hard. Independent JPEG frames or short GOPs take load off the decoder but bloat storage volume and traffic, and at the scale of hundreds of thousands of episodes that quickly becomes a real cost line item.

Two levers: sparse decoding and DDRA

The researchers attacked both causes at once by aligning the decoding schedule with the reference structure.

The first component, dependency-aware sparse decoding, reconstructs only the requested frames and the references strictly necessary to assemble them. Everything else is skipped. The source videos don't need to be re-encoded for this, so the method can be applied to datasets that have already been accumulated.

The second component, Dependency-Decoupled Random Access (DDRA), is optional and works as offline encoding. Each internal frame gets references to only two bounding intra-encoded anchors. Inter-frame prediction is preserved, so compression stays efficient, while long dependency chains disappear.

Together these techniques produce a hard bound: no more than three frames are reconstructed per requested frame (the target plus two anchors). This upper limit doesn't depend on the length of the intra-period, meaning the GOP can be made long for the sake of compactness without paying for it in decoding time.

What the measurements showed

Experiments ran on a large robotics dataset corpus, with the LeRobot v3 configuration serving as the baseline. Results for DDRA H.265:

• 58.7% less media data;
• 1.45x data-loading throughput;
• comparable frame reconstruction quality;
• comparable results on VLA, WAM, and VLM tasks given suitable compression settings (separate experiments by the authors).

The last point is the most important one in practice. A speed gain means nothing if the model trained on heavily compressed frames performs worse afterward. The authors separately verified this on three types of models and stress a caveat: this holds for suitable compression settings, not any arbitrary bitrate.

The authors include Zetian Song, Qize Yu, Yue Chen, Shiyu Huang, Siwei Ma, and Wen Gao. The project is tied to the LeRobot ecosystem and explicitly recommends against using its default video storage.

Who should run the benchmark

The first candidates to test this are teams training policies for manipulators and humanoids on LeRobot datasets. For them, video makes up the bulk of the data volume, and the loader often bottlenecks on decoding rather than on the GPU. If the 1.45x speedup holds up on their hardware and with their sampling pattern, every epoch gets faster.

The 58.7% volume savings matters independently of speed, too. It cuts costs for storage, for copying between clusters, and for loading from remote storage.

The sequence of steps is clear. Sparse decoding can be tested on existing data without re-encoding, and if loading speeds up noticeably, the next step is DDRA and offline re-encoding of the archive. Before that, it's worth running your own task set at different compression levels, as the authors did, and confirming that downstream model metrics don't drop.

The repository is still modest, but its openness makes it possible to compare the approach against a current pipeline using one's own data.