OmniXRef: A Unified Framework for Video Editing with Multimodal References

An editor's creative intent rarely fits into a single reference. OmniXRef is a new unified framework from Adobe Research for video editing that lets users combine an arbitrary number of text instructions, images, and video references. Each reference governs a distinct visual attribute — character motion, camera movement, style, special effects, or frame content. All references are applied to the source video in a single pass, producing a coherent edited result.

Why video references matter

Complex visual details and how they change over time — a character's physical motion, camera movement, the dynamics of visual effects — are hard to describe with text alone or a static image. A video reference lets creators precisely convey nuances of motion and visual transformation that the editing model must then reproduce in the context of the source video.

Architecture and data scaling

A unified architecture with a reference bank: every reference — whether text, image, or video — uses a shared interface consisting of an editing task definition, an instruction, and optional visual data. This lets the model process heterogeneous modalities in a uniform way.

OmniXRef-Model with a diffusion transformer: the model uses a single-stream diffusion transformer that processes the source video and all multimodal context as an interleaved sequence of tokens. Jointly encoding every reference in a single sequence creates synergy across different types of edits.

R²-RoPE for flexible length: a dedicated positional encoding mechanism, R²-RoPE, enables generalization to an arbitrary number of multimodal references rather than a fixed count, simplifying practical use.

Large-scale training with curriculum learning: the data synthesis pipeline produced over 3 million video-editing examples across different reference combinations and 7 million text-editing examples, for a total training dataset of roughly 10 million examples. The authors applied curriculum learning — progressively increasing task difficulty and video resolution as training proceeds.

OmniXRef-Bench and results

To systematically evaluate video as a control modality, the authors introduced OmniXRef-Bench, a benchmark of 125 editing cases spanning five key categories:

• Camera — controlling camera movement and viewing angle • Motion — controlling character physicality and movement • Content — changing frame content (replacing or adding objects) • Style — transferring visual style between videos • Visual Effects (VFX) — applying and integrating special effects

The benchmark measures three key metrics: editing fidelity (how well the model followed the instructions), source-video preservation (how much of the original video's information remains intact), and visual quality with temporal consistency checks (absence of flicker and artifacts across frames).

Results: OmniXRef outperforms task-specific baseline models as well as commercial solutions. The model delivers better results than Seedance 2.5 and MiniMax-H3 both in adherence to visual references and in preservation of the source video, confirming the effectiveness of a unified approach to multimodal editing.

Project page and details: https://oliver-cong02.github.io/OmniXRef.github.io/