Ensemble distillation on Kaggle: one model beat its teacher and saved 6.5 hours

In competitions with a strict inference limit, the bottleneck is often not quality but time. That is exactly what happened to the team that placed 19th in Kaggle's RSNA Knee Abnormality Detection competition: its ensemble of ConvNeXt, CoAtNet, and ResNet-GRU models took about seven of the nine permitted hours. There was simply no room to add other strong models.

How to compress an ensemble into one model

The solution was classic distillation, implemented with careful attention to detail. The teacher consisted of out-of-fold predictions from an early ensemble: each study was predicted only by models that had not seen it during training. This prevented label leakage from entering the student through the soft targets.

The targets used a 50/50 mixture of probabilities inferred by an LLM from radiology reports and the teacher's OOF probabilities. The student was a ConvNeXt-Tiny with a slice-attention head: a 140 mm field of view, 384 px input, eight windows per series during training, and 16 during inference. The setup also used an EMA of the weights, a pairwise ranking loss on top of BCE, and separate learning rates for the backbone and head.

The result: on the private LB, the student scored 95.1% versus the teacher's 94.8%, while inference took about half an hour instead of seven hours. Averaging six or 12 students added almost nothing (95.2%), so one model was enough.

Global and local views

The second part of the solution was a MedSAM branch. When an entire image is reduced to a small size, fine details disappear. The team therefore used one global model to examine the whole knee and five specialist models to inspect local crops for specific findings.

MedSAM was kept frozen and used without fine-tuning: it segmented the bone on the middle slice, and the upper boundary of the mask's largest component served as an anchor. Crops measuring 288×288 pixels, or about 80 mm, were placed at fixed offsets from that anchor. Blending the global model with the specialists produced 96.4% on the public leaderboard and 95.3% on the private leaderboard, while individually they scored 96.0/95.1% and 95.9/94.8%, respectively.

LLM as an annotator

The labeling process is particularly interesting. A fresh agent was launched for each study with the prompt "you are a radiologist." It read the report and DICOM data and returned probabilities for 12 findings, with instructions to use the full range from 0 to 1. The best result came from averaging two runs: one with the DICOM data and report, and another with only the DICOM data. Report-only labeling with a different model was noticeably worse. The team relabeled 440 disputed Lateral OA cases, but the authors could not measure the effect.

What happened with blending

The most practical section concerns blend weights. Fixed weights, per-finding weights tuned on a Gold set of 58 studies, and shrinkage toward equal weights differed by no more than 0.1 percentage points on the private LB. The more precisely the weights were fitted to Gold, the better the score on that set and the worse the public score. Split-half CV had already shown that this tuning was harmful.

What to take away

• If inference is constrained by a time limit, try distilling OOF predictions: the student may match or even beat the teacher.
• LLM-generated soft targets from textual reports are an inexpensive way to obtain probabilistic labels.
• Do not optimize blend weights on a tiny validation set: equal weights are almost always safer.
• For fine details, add local crops to a global model instead of increasing the resolution of the entire image.

The limitations are also worth remembering. The authors acknowledge that they ultimately trusted the public leaderboard, while the Gold set of 58 cases was too small to compare strategies reliably. This is a practical case study from a medical Kaggle competition, not a clinically validated system, but the recipe applies to any task where an ensemble does not fit within the inference-time budget.