SAM 2.1 fine-tuned on facades and interiors: mIoU rises from 49.9% to 69.6%

General-purpose segmentation models know little about construction. Researchers took SAM 2.1, fine-tuned it on the HBD dataset (2679 annotated images of 42 buildings), and raised mean mIoU across 11 building-element classes from 49.9% to 69.6%. That is a gain of 19.7 percentage points, or about 40% relative to the baseline result. The improved masks were then plugged into the GARField pipeline, which assembles 2D segmentation into a three-dimensional scene.

Why base SAM struggles with buildings

Reconstruction from photographs starts with understanding what is in each frame. To turn a set of images into a building model with separate walls, windows, columns, and beams, every object first has to be segmented in 2D. GARField handles the next step: it takes masks from Segment Anything Model and reconciles them across viewpoints in a neural radiance field (NeRF). The user gets a 3D grouping at a scale of their choosing: it can isolate an entire wall or a single panel.

The weak point is the quality of the input masks. SAM is trained on a broad set of images and does not distinguish engineering categories. It can slice a wall into pieces or merge a beam with the ceiling. When the output feeds a BIM model or measured drawings, that confusion is noticeable. So the authors fine-tuned the model on a domain-specific dataset with 11 classes: Wall, Window, Ceiling, Door, Column, Beam, Floor, Opening, Curtain Wall, Roof, and Lift.

What the measurements showed

Quality was scored with per-class IoU and Dice, plus overall mIoU. SAM 2.1 was then compared with SegFormer, which served as the baseline and generated masks by class. Finally, the pretrained and fine-tuned versions of SAM 2.1 were tested inside GARField, on freshly re-rendered images.

• Dataset: HBD, 2679 annotated images, 42 buildings.
• Classes: 11, from walls and windows to lifts and curtain walls.
• Metrics: per-class IoU and Dice, overall mIoU.
• Result: mIoU rose from 49.9% to 69.6%.
• Versus SegFormer: the fine-tuned SAM 2.1 wins on some classes, mainly those with large surface areas.
• In 3D: the fine-tuned version groups building elements better at the scale the user selects.

Caution is warranted here. The abstract gives no per-class breakdown, so it is unclear which categories improved the most. The SegFormer comparison also reports a win only on some classes with large areas. Small and thin elements such as beams or openings probably remain difficult, but that is an inference from context, not from the numbers. It is also unclear whether the dataset was split by building: if images of the same building landed in both training and test sets, mIoU is inflated. With only 42 buildings, this question matters.

Where this is useful

The main value of the work is that it shows a practical path for a niche industry. There is no need to train a segmenter from scratch: a few thousand annotated images are enough to make a foundation model noticeably more useful for a construction task. For companies that scan sites, monitor construction progress, or digitize existing buildings, this lowers the barrier to entry. A single set of annotations can improve the entire chain, from 2D masks to 3D grouping.

The practical workflow looks like this. First, build your own dataset for the classes you need. Then fine-tune SAM 2.1 and compare it with a lightweight model such as SegFormer: for large surfaces it may prove sufficient and cheaper at inference. After that, check the result not with 2D metrics but by the quality of the 3D grouping in GARField. That is where you can see whether building elements have become coherent objects.

Automatic reconstruction good enough for project documentation is still a long way off. But 69.6% mIoU against 49.9% is a measurable step, and it was achieved on real buildings, not synthetic ones.