OMNI: One Compact Encoder Absorbs the Knowledge of 10 Models for Pathology

A team of researchers has published Scaling Multi-Teacher Distillation for Digital Pathology on arXiv. The paper shows how to compress the knowledge of ten state-of-the-art digital pathology foundation models into a single compact encoder. The result is called OMNI. According to the authors, these students are up to 10 times more efficient than the large teacher models. They are comparable to the teachers in quality and outperform them on a significant share of tasks. The evaluation covered more than 10 foundation models and 39 tasks at the level of individual tiles and whole slides.

Where multi-teacher distillation breaks down

Digital pathology has recently seen a wave of strong foundation models. Each is trained on its own data with its own recipe, and each has its own strengths. The idea of multi-teacher distillation (MTD) suggests itself: gather several teachers and transfer their knowledge to a single student network. Then you no longer have to run a dozen heavy encoders on gigapixel images.

In practice, it works worse than that. The authors show that standard MTD responds poorly to a growing number of teachers. Adding experts pays off only up to a point; beyond it, the student does not get a proportional gain. The paper's diagnosis is that the key to scaling is learning teacher-specific representations. If you mix all the teachers into one shared space, their differences get erased, and those differences are exactly where the ensemble's value lies.

Four fixes instead of one

To let the student preserve each teacher's particular strengths, the authors change both the architecture and the loss function. Each teacher gets its own learnable tokens, which give the network a separate "entry point" for each expert. The attention scheme is tuned specifically for this design. The encoder gains additional mixture-of-experts layers that route computation to specialized branches. A contrastive loss further separates the representations of different teachers.

As a result, the method scales better with the number of teachers. This is exactly what baseline MTD failed to do.

What was tested and what was released

Key parameters of the work:

• 10 state-of-the-art foundation models used as teachers;
• more than 10 foundation models evaluated in the comparison;
• 39 tasks, including tile-level and whole-slide-level;
• students up to 10x more efficient than the large teachers;
• quality on par with the teachers, and in places higher.

The authors are releasing the OMNI collection, a set of efficient foundation models trained on the knowledge of ten teachers. Separately, they analyzed the learned representations and found them to be complementary, meaning they carry different information. The authors cite this as the reason the representations are easy to aggregate at the downstream stage.

What to change in practice

For teams building pathology pipelines, the main practical takeaway is cost. Slide analysis is compute-bound: a single slide is split into a huge number of tiles, and each one is passed through the encoder. A model that is ten times more efficient at comparable quality directly lowers inference cost and lets you run processing on more modest hardware.

The second takeaway is methodological. If you distill from several models, you should not merge them into a single target vector. Separate tokens, MoE layers and contrastive separation will likely be useful outside pathology too, in any domain with many strong specialized models. This is still an assumption; the paper did not test it.

The limitations deserve attention. This account is based on the preprint's abstract, so detailed per-task metrics need to be checked in the paper itself. The licensing terms for the OMNI weights and the results on your own data must be verified separately. For clinical use, benchmark accuracy alone is not enough: it requires validation on a specific lab's data and regulatory procedures.