L3G-World Generates 3D Worlds in Real Time at 9.02 Frames/s on an H100

Interactive world models generate video in response to user actions, but they struggle with long walks through a scene: return to a place you have already seen, and it looks different. The authors of L3G-World propose storing the scene not as a history of frames but in compact memory made of 3D Gaussians, each with an attached VAE latent feature. According to the authors, the system builds a spatially consistent world from a single image and a camera trajectory, and runs at 9.02 frames/s on a single H100.

Why both conventional memories scale poorly

The first approach uses temporal context: when generating the next chunk, the model looks back at past frames or at the KV-cache of its self-attention layers. The cost of this memory grows with stream length and resolution, and over long distances the model runs out of GPU memory. In their comparison table, the authors report OOM cases for such methods when generating 400 frames on an H100.

The second approach stores the scene explicitly, as a point cloud or as Gaussians. This memory is tied to space, so it holds consistency better when the camera returns to earlier places and gives precise camera control. But it has its own trade-offs. Feed-forward methods predict primitives for every pixel and reconstruct a scene quickly, but the primitives pile up repeatedly in the same regions. Memory bloats, rendering slows down, and nothing corrects inaccurate geometry. Per-scene optimization methods are more compact, but iterative fitting is too slow for real time.

There is a second problem. Explicit 3D memories usually feed the generator an RGB render, which has to be run through a VAE encoder again. That step alone rules out real-time operation.

How L3G-Mem decides where to add Gaussians

The system has two modules that alternate autoregressively. L3G-Mem updates the memory, and L3G-Gen generates the next video chunk. Memory grows mainly with scene coverage, not with the number of generated frames.

Updates happen in windows. First, the current Gaussians are rendered from the new cameras, and pixels with low opacity or a large RGB error go into a generation mask. For these regions, the system takes features from a geometric foundation model and clusters them with K-means (following the ZipSplat scheme) into a small set of queries. That way, the budget for new Gaussians goes only to what is not yet in memory.

Meanwhile, the visible Gaussians are refined. Their memory tokens are extracted as refinement tokens. A single transformer processes both streams. In each block, the streams first attend to geometric features, then go through self-attention within the stream and shared global self-attention. New Gaussians are added to memory, and old ones are updated with residual corrections. As a result, the memory does not turn into a buffer: geometric errors from earlier steps are gradually corrected.

A separate branch lifts VAE latent features into 3D. Thanks to them, the memory renders straight into the generator's latent space, and no re-encoding is needed. Training runs over a sequence of windows with a stop-gradient through the previous memory. The loss includes RGB, LPIPS, depth, MSE on VAE latents, and a visibility penalty.

The L3G-Gen generator and what remains unclear

L3G-Gen is a video DiT with autoregressive generation. Latents rendered from memory and an opacity mask are concatenated with the noisy latents along the channel dimension, meaning no extra tokens. Previously generated anchor latents are fed in separately and stay unchanged. Self-attention uses RayRoPE, a residual positional encoding based on camera rays, and the anchor latents get a shifted temporal RoPE. The teacher is trained with rectified flow, and for speed it is distilled into a generator that needs only a few steps.

The authors claim the best results in generation quality and long-range self-consistency. They also report fewer 3D primitives, faster memory reconstruction, and faster rendering of VAE features compared with the most efficient prior baseline on each metric. The exact multipliers are cut off in the available text, so check them against the original paper. The comparisons were made by the authors themselves, and there are no independent verifications yet.

The practical takeaway for anyone following world models: 9.02 frames/s on an H100 is not yet 24–30 FPS, and that speed comes from a top-end accelerator. But the architecture shows how to build scene memory whose cost does not grow with session length. That matters for game simulators, training agents in synthetic environments, and interactive generation.