RLSplat teaches 3D Gaussian Splatting to economize: up to 4.1× faster and 9.6× less memory

Feed-forward 3D Gaussian Splatting builds a three-dimensional scene from a set of photos in a single neural network pass, without lengthy per-scene optimization. RLSplat adds something its predecessors lacked: latency, peak memory and representation size all fall linearly with the specified Gaussian budget. In zero-shot tests it outperforms earlier budget-controllable approaches at every budget and beats the strongest baseline even with four times fewer Gaussians. At small budgets it runs up to 4.1× faster and uses 9.6× less peak memory.

Pruning that eats all the savings

The idea behind budget control in FF3DGS is straightforward: the output scene has to fit within a limit on storage or rendering time. A phone or headset needs a lightweight scene, while a server with resources to spare can handle a dense one. Existing methods offer this kind of control, but with a catch. To get, say, 50 thousand Gaussians, they first predict a dense set and then select or cut away the excess.

The result: the final file really is small, but getting there costs just as much. Inference latency and peak memory stay almost constant regardless of the budget. For a device whose problem is specifically memory during inference, such a "budget" solves nothing: the resources for the dense stage are still required.

How RLSplat allocates Gaussians

The authors remove the "many first, fewer later" stage entirely. Instead of pruning or selection, RLSplat trains an allocation policy. It distributes the given budget across local regions of the scene, and the output contains exactly the requested number of Gaussians. Complex areas get more, simple ones get fewer.

The second part of the solution concerns parameter refinement. Gaussians pass through resource-linear Gaussian decoding: each one attends only to its nearest neighbors, which an efficient hierarchical search finds. Full attention across all Gaussians would grow faster than the number of elements itself. Local attention makes decoding cost linear, so both time and memory fall as the budget shrinks.

Key characteristics from the paper:

• Budget: the number of Gaussians is set at inference and produced exactly, with no post-selection.
• Scaling: representation size, latency and peak memory fall linearly with the budget.
• Speed: up to 4.1× faster than previous methods at small budgets.
• Memory: up to 9.6× lower peak memory at small budgets.
• Quality: better than earlier budget-controllable methods at every budget in the zero-shot evaluation.
• Efficiency: outperforms the strongest baseline while using 4× fewer Gaussians.

Where this can be applied

The result matters most for scenarios with a hard resource ceiling: AR and VR on standalone headsets, mobile apps for quick room scanning, robotics, where a map must be updated on board, and cloud services that process thousands of scenes in parallel. In the last case, a nine-fold saving in peak memory means more jobs fit on a single accelerator.

The takeaway for engineers is practical. If you are choosing a reconstruction pipeline for a memory-constrained device, look not only at the size of the final scene but also at how latency and peak consumption change as the budget drops. Approaches that select from a dense set look weaker on this test. An approach that allocates directly for the target budget, as RLSplat does, lets you tune quality to the hardware smoothly: reduce the budget and you get a proportionally faster and lighter result.

The fact that the gain comes in zero-shot mode, that is, without fine-tuning on new data, makes the method a candidate for products where scenes are not known in advance. It remains to be checked how the allocation policy behaves on very large or atypical scenes, but the principle is already clear: inference cost should scale with the requested output.