ComfyUI adds RelayVSR: real-time video upscaling to 4K with 14 GB of VRAM

What problem does it solve?

Anyone who has tried enhancing video with generative models knows the pain: the quality is excellent, but every frame is processed as a separate image. This is slow, and details begin to flicker between frames. Conventional fast upscalers have the opposite problem: they work quickly but produce a blurry image without adding new detail.

RelayVSR, now supported in ComfyUI, tries to resolve this trade-off. The idea is as simple as a relay race: the heavyweight model runs only the opening legs, then hands the baton to a lightweight one.

How it works

The architecture is called Sparse Generative Relay. A large generative model synthesizes detailed latent references only for sparse keyframes—for example, one frame out of every few dozen. A lightweight transformer with dual memory, the Dual-Memory Video Transformer, then propagates these details across every intermediate frame in the stream.

Imagine an art restorer and an assistant. The restorer carefully reconstructs the character's face in one frame. The assistant does not redraw it from scratch, but carefully carries that work into adjacent frames while accounting for motion. Dual memory is needed to retain both the latest context and the reference frames.

One distinct problem with this approach is that errors accumulate. If an artifact appears in one frame, it propagates further. To address this, the authors use Video-Aware Reference Optimization (VARO) reinforcement learning, which teaches the system not to accumulate defects over long sequences.

Numbers and hardware

The claimed results depend on quantization. The available INT8 weights are relayvsr_x4_int8_convrot and relayvsr_x2_lightx2v_int8_convrot. With them, the system delivers 30 frames per second at 1080p on a single consumer GPU while using around 14 GB of VRAM. In practice, that means a GPU with 16 GB or more.

There is an important caveat, however. Upscaling to 1080p and 4K is possible, but the description indicates that true real-time processing requires an A100. So the claim of "real time on a consumer GPU" holds for 1080p; 4K requires server-class hardware or offline processing.

What to do with it

If you build video services, streaming systems, archive-processing pipelines, or editing tools, it is worth trying. The models are already available from the Comfy-Org organization on Hugging Face, the authors have a page at kopperx.github.io, and the system can be run in Colab notebooks without the hassle of a local installation.

Practical advice:

• Test your own footage for detail flicker and behavior in fast scenes and at shot changes: these are precisely where sparse keyframes may fail.
• Compare the x2 and x4 weights: 2x upscaling is sufficient for many tasks and is likely to be more stable.
• Measure actual performance on your GPU instead of relying solely on the announcement's figures.

Limitations

For now, these are the authors' claims and a recent integration. The article contains no independent benchmarks, and nothing is said about performance on difficult content such as text, fine graphics, or abrupt motion. Generative upscaling also invents details by design, which creates a risk for documentary or forensic use: the image may become more attractive without becoming more accurate.

The bottom line: if the combination of "infrequent expensive frames plus cheap propagation" proves capable of preserving quality, it could become a useful pattern for other video tasks as well. RelayVSR is already worth testing for anyone constrained by upscaling speed, and it is one of the few cases in which a generative approach is moving closer to real-time performance.