OmniXRef Combines Text, Photos and Video in a Single Video Editor

The OmniXRef team has published a unified video editing model that accepts three types of control signals in a single request: text, images and other videos. The number of references is not fixed. Quality is tested on a new benchmark, OmniXRef-Bench, across five categories of attribute transfer: camera, motion, content, style and visual effects. The authors write that the model outperforms both specialized baselines and commercial systems. The abstract gives no specific metrics, so the numbers will have to be found in the paper itself.

The idea targets a familiar problem with diffusion-based video editors. A text command like "remove the people" works well. A reference image helps with an object's appearance. But a phrase like "shoot it the same way as in that clip" can't be conveyed in words: a camera trajectory, the rhythm of movement or the character of an effect is easier to show with another video. This control channel is usually the least developed in existing tools.

How the model works

At its core is the reference bank interface. It is a single container that holds all of a request's references, regardless of modality or number. To keep the model from breaking when there are more references than in the training examples, the authors proposed R^2-RoPE positional encoding. By their account, it supports generalization to a larger number of references.

OmniXRef-Model itself is a single-stream diffusion transformer. The source video and the multimodal context prepared by an MLLM are fed into it as one interleaved sequence. There are no separate branches for each type of condition. This makes it easier to add new input combinations, although the computational cost depends on sequence length, and the authors do not disclose it in the abstract.

Data and benchmark

The main bottleneck here is data. Natural triplets of "source video, video reference, result" barely exist. So the team built OmniXRef-Data with synthetic pipelines, separate for different editing tasks and for their combinations with references. Training followed a curriculum learning scheme: simple tasks and low resolution first, then rising difficulty and resolution.

OmniXRef-Bench fills a gap in evaluation. Until now, video as a control channel had barely been measured systematically. There are now five comparison axes:

• Camera: transfer of camera movement;
• Motion: transfer of object movement;
• Content: replacing or adding frame content;
• Style: transfer of visual style;
• VFX: transfer of visual effects.

Quick specs based on the abstract:

• inputs: text, images and video in variable numbers;
• interface: reference bank;
• positional encoding: R^2-RoPE;
• architecture: single-stream diffusion transformer plus context from an MLLM;
• data: synthetic OmniXRef-Data;
• training: curriculum by difficulty and resolution;
• evaluation: OmniXRef-Bench, 5 categories;
• claimed result: better than task-specific and commercial models, with no numbers in the abstract.

What is still unknown

The description does not give the parameter count, maximum clip length and resolution, generation speed or license. The commercial models used in the comparison are not named either. There is also a methodological risk: the benchmark was created by the same authors, and the training data is synthetic. That combination often wins specifically on its own labeling. An independent run on other people's footage will show the real level better than any table.

How to apply it

A team that makes ads, music videos or social media content should wait for the code and weights to be released, if the authors publish them, and run the model on its own material. Camera and motion transfer deserve checking first: that is where a video reference has an edge over text. To compare against commercial services, it is convenient to take the five axes from OmniXRef-Bench and score each one separately rather than going by overall impression. Editor developers should look closely at the reference bank idea: a single interface for different references is easier to build into a product than a set of separate modes for each type of condition.