EgoX Merges Camera, Hands and Objects Into One 3D Scene, With 65.6% Lower Hand Error

The EgoXperience team has proposed a way to reconstruct human activity from first-person video so that the observer's trajectory, the room geometry, the hands and the objects in them do not contradict one another. The EgoX model takes a monocular clip and object masks in the first frame. In a single forward pass, it produces a metric 3D reconstruction of the camera, scene, hands and objects in a shared world coordinate system. Compared with the strongest baselines the authors tested, camera error is 68.9% lower, hand error 65.6% lower and depth error 42.9% lower, while object pose estimation improved by 28.0%.

A problem of consistency, not of individual components

Egocentric video is hard for an obvious reason: a head-mounted camera is constantly moving, hands occlude objects, and the scene is visible only in fragments. Existing approaches usually tackle a single subtask. Some estimate camera pose, others reconstruct hands, and still others handle depth or object pose. When you stitch the results together, they often fail to line up: a hand "floats" in the air relative to the table, or an object drifts away from the camera's trajectory.

The authors argue that consistency requires two things: joint reconstruction and aligned annotation. In their assessment, current methods and datasets support both poorly. So they built not just a model but the whole pipeline: data collection, automatic annotation, a unified model and a benchmark.

How the pipeline works

The core component is EgoX-Engine, an automatic annotation engine. It links geometrically verified object tracking with SLAM-based alignment to world coordinates. As a result, the camera, scene, hands and objects are annotated in one coordinate system and are consistent with each other. The annotation requires no manual work on each frame, which is what makes such a dataset affordable to build.

The result is the EgoX-Data-6M dataset: 50 hours of video with task-oriented manipulation and movement indoors and outdoors, including transitions between the two. The clips come with task captions accurate to the second. From this collection, the authors selected one hour spanning different activity categories and packaged it as EgoX-Bench for evaluating joint reconstruction. The EgoX model itself was trained on EgoX-Data-6M and on additional annotations from other public datasets.

Key specifications and results

• Input: monocular egocentric video and object masks in the first frame.
• Output: metric reconstruction of the camera, scene, hands and objects in a single world coordinate system.
• Mode of operation: a single forward pass, with no separate optimization stages mentioned in the abstract.
• Dataset: EgoX-Data-6M, 50 hours, manipulation and movement indoors and outdoors, captions accurate to the second.
• Benchmark: EgoX-Bench, one hour of video across different activity categories.
• Camera error: 68.9% lower than the strongest baselines tested.
• Hand error: 65.6% lower.
• Depth error: 42.9% lower.
• Object pose estimation: 28.0% higher.

Where this could be useful

The first audience is robotics. Data on how people pick up objects and move through space is needed to train manipulation from demonstrations. If hands and objects can be extracted automatically in metric coordinates from ordinary first-person video, the barrier to collecting such demonstrations drops noticeably. The second audience is AR and smart glasses, where the system needs to understand the scene and the user's actions relative to real space.

The limitations are also clear. The model needs object masks in the first frame, so it cannot yet be called a fully autonomous system. A 50-hour dataset is modest by the standards of video models, and a one-hour benchmark gives only a limited picture of generalization. The figures come from the authors themselves and apply to the baselines they chose. For practitioners, the main questions are whether the code, weights and data will be released, and how well the results hold up on other people's capture devices. If they do, EgoX-Bench could become a convenient shared point of comparison for joint reconstruction methods.