SynthAVE: when 21 models labeled 12 thousand products instead of humans

Researchers published SynthAVE—a method for mass synthetic labeling of e-commerce products using multiple LLM voting. Instead of manual annotation of millions of data points, the team applied a network of 21 LLM configurations (7 models × 3 prompts), whose results agree with expert human assessments at 95.2% (Cohen's κ = 0.92).

Why Manual Labeling No Longer Scales

In e-commerce, each product has dozens of parameters: size, color, material, weight, sleeve length, chest size, and so on. A major marketplace works with thousands of product types and hundreds of attributes simultaneously. This means full labeling requires millions of annotations. Hiring humans for such a volume costs tens of millions of rubles per month.

Companies have long known that LLMs can generate labeling automatically. But the question arises: how do you verify quality at such scale? Manual verification degenerates into the same expensive work.

Method: Arena of Voting Models

Instead of using a single LLM, the authors invoke 7 different models, each with 3 prompt variants. Each product sample is evaluated by all 21 configurations independently, then the final result is selected by majority vote.

Key dataset figures:

• 12,726 products
• 229 product types
• 792 different attributes
• 4 languages: Spanish, French, Italian, German

Result: Human-Level Performance at Machine Speed

Majority voting achieved Cohen's κ = 0.92 against human labeling (95.2% direct agreement). Individual models achieve κ = 0.76 with each other.

The paradox: each model has its own blind spots. One understands fabrics better, another sizes, a third materials. But when many models vote, their errors cancel each other out—like a good jury.

What This Means

e-commerce companies gain the ability to scale data labeling to human level at 10–100 times lower cost than manual work. This opens the door to automating thousands of microtasks that previously required people. The method is universal: anywhere massive labeling is needed (medicine, finance, logistics), you can assemble a panel of judges and use voting.