Habr author fine-tunes YOLOv8l on dental X-rays and pushes mAP50 to 0.642

A Habr author has published the second part of the "AI Dentist" project. The pathology detector for panoramic dental X-rays has moved from YOLO26n to YOLOv8l (209 layers, 43.6M parameters, 165.4 GFLOPs). The final two-stage training run at 1280×1280 resolution took 29.3 minutes on a single NVIDIA A100 80GB PCIe. On the DENTEX Challenge validation set (144 images, 709 objects), the model scored 0.642 mAP50 and 0.408 mAP50-95. The value of the work lies not in a record score but in an honest account of what worked and what didn't.

Data cleaning and protecting validation from leakage

In the first part, the prototype was tested on four classes, but it turned out that color photos of the oral cavity had slipped into the training archive. They caused detections outside the target distribution. An intermediate run on nine classes (including fillings, crowns, and implants) gave a mAP50 of 0.511 and an AP50 of just 0.069 for periapical lesions. Fillings and root canal fillings made up more than 65% of all labels, while there were only 1 584 lesions. The restorations were removed, leaving four classes: Caries, Deep Caries, Impacted, and Periapical Lesion.

The Clean_Dental_v3 set combines DENTEX (2585×1353), the Tufts Dental Database (1 000 images), and the Loki_Dental repository (13 932 files). Validation uses DENTEX only. Under the dataset's rules, one image corresponds to one patient, so there is no overlap between train and val. External datasets went exclusively into training. Deduplication with dHash removed 33 duplicates, and a filter on aspect ratio and color channels discarded 1 571 images.

Two bugs in the dataset builder were also fixed. Previously, large Loki_Dental boxes were automatically turned into Deep Caries based on area (2 463 objects), even though caries depth is defined by proximity to the pulp chamber, not by area. Now Deep Caries consists of 565 DENTEX annotations, and all Loki caries stays in the Caries class (3 556 objects).

In addition, Loki reading stopped at file 6 118 of 13 932 because of a 1 200-image limit, so 56% of the archive was lost. A new two-pass selection found 2 379 candidates with periapical lesions and cysts. Rare classes were duplicated in the training list: 3x for DENTEX images with Deep Caries and Periapical Lesion, and 2x for periapical images from Tufts and Loki.

Two training stages and three failed tricks

The first stage started from the pretrained yolov8l.pt with all layers active: 25 epochs in 23.9 minutes, stopped by EarlyStopping, with the best result at epoch 10. The second stage ran 12 epochs in 5.4 minutes with the first 10 backbone layers frozen, lr0=0.0001, and a slightly higher classification loss weight (1.8 versus 1.6). Preliminary debugging on a Tesla T4 in Colab took about three hours. Stage by stage, mAP50 rose from 0.609 to 0.642, and AP50 for periapical lesions rose from 0.461 to 0.516. Per-frame time on the A100 is 5.8 ms.

Two "free" improvements were also tested. Linear averaging of the weights from the two stages (Model Soup 50/50) gave a small gain on the T4 at 1024 px, but at 1280 px it lowered mAP50 to 0.634.

Precision at the maximum-F1 point did rise from 69.6% to 75.6%, but per-frame time rose to 7.

3 ms. The production version uses the clean checkpoint from the second stage. Multi-scale TTA in Ultralytics made results worse: precision for Deep Caries fell from 60.

5% to 25.9%, and for Periapical Lesion from 39.2% to 20.

3%, while per-frame time on the T4 grew from 54.5 to 128 ms. The author attributes this to running a downscaled copy (0.

83x) and merging low-threshold boxes before NMS. That hypothesis was not tested separately.

The single conf=0.50 threshold was replaced with per-class thresholds taken from F1-Confidence curves: Caries 0.23, Deep Caries 0.21, Impacted 0.56, Periapical Lesion 0.26. Thresholds are higher for compressed images from the internet and lower for screening.

Where the model fails and what blocks clinical use

Impacted performs best: F1 0.914, precision 92.3%, recall 90.6%. Periapical lesions are weaker: precision is 76.4%, but recall is only 43.3%, meaning more than half of the lesions are missed. For screening, that matters.

On the control panoramas, the model found all annotated regions in one case and 5 of 6 in another, but it missed caries on an upper right molar. It also added boxes that are absent from the DENTEX annotations (teeth #48, #36, #25, #26). Without blind review by radiologists, there is no way to say whether these are missed pathologies or false positives. There is also a systematic error: FDI tooth numbering is computed from the box center relative to the line y=0.52, so caries on an upper premolar was labeled #45 instead of #15. The author plans to replace this formula with separate segmentation of all 32 teeth.

The author names the practical limitations himself. Metrics were measured on DENTEX only, and Russian clinics need a test set from domestic orthopantomographs. On compressed images from messaging apps (650×340–900×600), small caries is detected less reliably, so the interface needs a low-resolution warning. The main takeaway for practitioners: on a small medical dataset, careful annotation work, isolated validation, and threshold calibration delivered more than changing the architecture or using tricks like TTA and weight averaging. The entire training run fit into half an hour on a single card.