TAPe Lifts Classification to 77% and Compares Itself with YOLO on a Small COCO Subset

The eighth entry in the TAPe diary is not about a single feature. It assembles nearly a complete detection pipeline: the authors improved segmentation, pushed classification to 77%, and tested what happens when the approach is compared with YOLO on a small COCO subset. Their interim conclusion is optimistic: the model already starts to work in a scenario where classic detectors usually need considerably more data.

Segmentation along the boundary

The main update at this stage is segmentation by contrasting patches on the real boundary of an object, rather than by a nominal box around it. The idea is to split the image into more natural segments and then assemble the object from them, instead of trying to guess a box straight from raw pixels. According to the authors, this quickly improved how well patches are merged and made it possible to form more adequate segments for each object in the image.

In parallel, the team tried several other architectural solutions: additional heads, different ways of selecting similar segments, and more complex aggregation schemes. These approaches did not deliver the expected effect. The reason, as the authors describe it, is that such schemes tried to fit TAPe data to a familiar architecture instead of using it as it is. In practice, a more direct path worked better: relying on the structure of the TAPe representations themselves and improving the links between patches.

Classification without a learning rate

The next problem turned out to be more down-to-earth: some patches sometimes failed to land in the right segment. If one or several fragments of the image are not associated with an object, they are hard to classify correctly, because the model cannot answer the question of what exactly that piece belongs to.

To bring training closer to the model's real behavior, the authors began simulating, during training, the step-by-step growth of a segment from a single patch, that is, repeating the same logic used at inference. This helped find the right links between patches, but did not remove every limitation. The remaining problem is "non-growing" regions, where a segment fails to gather enough context and therefore makes classification errors. For such cases, TAPe now additionally checks neighboring areas and smooths the context.

Separately, the authors describe another important engineering goal: consistently eliminating hyperparameters that can break the system's behavior. One of them was the learning rate, which this version drops together with gradient descent.

• Segmentation now works on contrasting patches at the object boundary
• Classification rose to 77%
• Dropping the learning rate added about 3% accuracy
• The weakest spots so far involve small segments and a lack of context
• The team's next goal is to reach at least 80% classification

The authors also note a market reference point: publications on DETR cite classification accuracy of around 79%, although it is not entirely clear whether detection errors themselves are included. For TAPe, this is not yet the finish line but the nearest bar. Full tests on the entire COCO dataset are still ahead because they take a long time, but it is already clear that the architecture has become more stable and is better tied to self-supervised learning tasks.

First tests against YOLO

The most striking part of the entry is the first direct benchmark against YOLO. For the experiment, the authors took a small COCO slice of 5 thousand images and split it 70/30: 3,500 frames for training and 1,500 for testing. For standard detectors, this volume turned out to be critically small. TAPe states that YOLO practically does not converge on this set, and its detection level stays at around 1%.

"YOLO does not converge at all on the dataset we use for tests."

This is not yet the final comparison on mAP50, mAP50-95, speed and parameter count; the authors are still preparing a separate post with full benchmarks against YOLO and RF-DETR. Even so, this early result matters because it shows the project's main thesis: TAPe aims to be not just another detection model, but an architecture that can work with tens of images per class where more familiar approaches need hundreds of thousands of examples and a much heavier pretrained base.

What this means

If TAPe does confirm its results on the full set of metrics, it will be a strong argument for computer vision that emphasizes data structure, not just scale. For teams with small datasets this is especially important: the entry price for quality detection may drop noticeably.