HiP3R assembles 3D scenes from thousands of video frames without retraining models

Researchers at Nanyang Technological University Singapore have published HiP3R, a method for globally assembling 3D reconstructions from long videos. It takes sequences of thousands of frames as input and returns a consistent camera trajectory and scene geometry. No fine-tuning or replacement of the base model is required: HiP3R works on top of existing neural reconstruction networks. The authors are Zunnan Xu, Shangchen Zhou, Yihang Luo, and Chen Change Loy. The paper was released on October 22, 2026, with the code made publicly available.

Why long video breaks feed-forward reconstruction

Feed-forward models have in recent years learned to estimate camera motion and 3D geometry from a short window of frames quickly and accurately. The problem starts when there are thousands of frames. Memory and compute requirements grow to the point where processing the entire sequence in a single pass becomes impossible. What's left is slicing it into overlapping chunks.

Slicing comes at a cost. Each chunk is reconstructed independently, so they end up with different coordinate systems and different scales. Frames in the overlap zone receive two separate pose estimates, and they don't match. To assemble a coherent scene from this, constraints need to be propagated across the entire sequence — from neighboring overlaps to loop closures, where the camera returns to a previously seen location. The classic solution for this task is pose graph optimization.

The dilemma of two graph levels

The level at which the graph is built determines what you end up with. Each option has its own trade-offs.

• A chunk-level graph. Effectively eliminates scale drift and closes large loops. But since chunks move as rigid bodies, it cannot fix internal errors or frame-level predictions that contradict each other.
• Frame-level optimization. Provides local flexibility. But global scaling becomes computationally expensive, and metric scale can break down in the process.

HiP3R resolves this trade-off through hierarchy rather than picking one side.

Two steps: first the skeleton, then the trajectory

In the first step, HiP3R builds a robust chunk-level graph using Sim(3) transformations — accounting for rotation, translation, and scale. This graph sets the global layout of the scene and the overall scale.

In the second step, the layout is frozen. A single shared camera in SE(3) — that is, pose without scale — is estimated for each frame. Overlapping predictions from neighboring windows are turned into a single trajectory instead of a set of duplicates. Since the scale is already locked in at the top level, frame-by-frame fitting doesn't break it. The coarse skeleton stays rigid, while details are refined where they're needed.

The authors tested the method on several benchmarks. According to their data, it improves both camera trajectories and 3D reconstruction quality. It is applicable to two types of backends — chunk-based and streaming.

Technical parameters

• Task: globally consistent 3D reconstruction from video sequences of thousands of frames.
• Level 1: a robust chunk graph in Sim(3) sets the layout and metric scale.
• Level 2: a single shared SE(3) camera per frame with the layout fixed.
• Compatibility: chunk-based and streaming backends.
• Training requirements: no fine-tuning needed.
• Organization: Nanyang Technological University Singapore.
• Authors: Zunnan Xu, Shangchen Zhou, Yihang Luo, Chen Change Loy.
• Publication: arXiv 2610.31635, project page at kkakkkka.github.io/HiP3R.
• Repository: GitHub, 361 stars at the time of this review.

Who will find this useful in practice

HiP3R works as a post-processing layer, so it can be plugged into an already functioning reconstruction pipeline. This is relevant for teams that shoot long clips with drones, wearable cameras, or robots and assemble maps from them. The same applies to those building 3D scenes for simulations, games, and training embodied agents. These use cases need a single consistent trajectory spanning kilometers of video, rather than a collection of unstitched fragments.

Developers with an existing streaming or chunk backend should test the repository on their own data. Priority checks include: scale drift over long routes, behavior on loop closures when the camera returns to previously seen locations, and trajectory stability in chunk overlap zones. The lack of fine-tuning lowers the barrier to entry — all that's needed is to connect the method to existing predictions. A repository with 361 stars shows that the community has already taken notice.