Spotify: an LLM-powered annotation platform — a 10x larger labeling corpus with 3x annotator productivity

• How We Generated Millions of Content Annotations — Spotify Engineering (Puleo, Seetharam, Drzyzga), 2024-10-21
• Spotify: Scaling ML Annotation Platform with LLMs for Content Classification (сторонний критический разбор) — ZenML LLMOps Database

Background

Spotify is the largest audio streaming service, with a catalog of hundreds of millions of tracks and podcast episodes. From a machine learning standpoint such a catalog is both an asset and a liability: recommendations, search, automatic placement of content on artist pages, detecting relations between releases, and moderating podcasts for policy violations — all of these are ML models that must cover the entire catalog. And any model, before it can work, needs labeled data — both for training and for quality evaluation.

Labeling is the most underrated part of the ML pipeline. While the industry debates model architectures, production teams hit the prosaic question: who will put millions of labels on audio, video, and text, how fast, and how reliably. Generative AI sharpened the problem twice over: model iteration cycles shrank from months to weeks, making labeling that can't keep pace the main brake — while LLMs themselves, for the first time, made large-scale automatic labeling of acceptable quality possible.

In October 2024 Spotify's annotation platform team — platform ops manager Dana Puleo and staff engineers Meghana Seetharam and Katarzyna Drzyzga — published 'How We Generated Millions of Content Annotations' on the engineering blog: how the company industrialized data labeling by combining tiered human expertise with an LLM system running in parallel with people. The post is notable for its genre: not the story of one model but a description of an internal factory platform serving dozens of labeling projects across the Music, Podcasts, and Audiobooks verticals — with an emphasis on catalog safety and moderation.

Problem

Data labeling at Spotify's catalog scale was the bottleneck of the entire ML pipeline. Content: hundreds of millions of items; tasks: dozens, from content classification and release-relation detection to analyzing podcast episodes for policy violations; and manual annotation is slow, expensive, and scales poorly. The classic dilemma: hire more annotators and costs grow linearly while quality, absent solid processes, even drops; automate naively and label quality becomes unpredictable — yet both training and honest model evaluation depend on it.

The second problem was fragmentation. Different Spotify teams built their own scattered annotation processes: their own tools, their own guidelines, their own quality criteria. Expertise wasn't reused, quality was uneven, and launching every new labeling project meant building a process from scratch — unacceptably slow for an era in which generative models demand fresh evaluation datasets every few weeks.

The third layer is specific to an audio platform: labeling sound and video is harder than text. An annotator must listen to segments, compare track versions, review podcast episodes — standard text-labeling tools aren't built for that. And the stakes are high: labeling errors in catalog-safety and moderation tasks convert directly into reputational and regulatory risks for the platform.

Solution

Spotify built a unified annotation platform on three pillars: scalable human expertise, flexible annotation tooling, and shared infrastructure integrated with ML pipelines.

The human side is organized in three tiers. Core annotators are domain experts providing first-pass review of all annotation cases. Quality analysts are top-level domain experts acting as the escalation point for all ambiguous or complex cases. Project managers connect the teams, maintain training materials, and run the feedback loop between labeling customers and annotators. The pyramid solves two problems at once: volume (first-pass labeling is cheap and fast) and quality (contested items flow up to the strongest experts).

In parallel with the humans runs a configurable LLM-based labeling system — the platform's key architectural decision. The model labels the same data as people; inter-annotator agreement is computed automatically, and items with low consensus — between humans, or between a human and the model — escalate automatically to quality analysts. In the team's words, this is what allowed Spotify to 'significantly grow our corpus of high-quality annotation data with low effort and cost': the LLM absorbs the bulk routine while scarce human expertise concentrates where it is irreplaceable.

The tooling is built for real content types: beyond standard interfaces for NLP tasks, the platform provides custom interfaces for annotating audio and video segments. Real-time project dashboards show completion rates, volumes, and per-annotator productivity — labeling turns from an 'outsourced black box' into a managed production process with a measurable SLA.

Finally, the integration layer: the platform is embedded in Spotify's ML workflows at every stage — through CLIs, UIs, and batch orchestration systems. Labeling results don't sit in spreadsheets; they flow back automatically into model training and evaluation pipelines, and a new labeling project is configured on top of ready infrastructure rather than built from scratch.

Result

The annotation corpus grew 10x and annotator productivity 3x: automating sampling, review serving, and feeding results back into training pipelines removed the manual steps, while LLM labeling took the bulk routine off people. The platform runs dozens of labeling projects in parallel, serving ML and GenAI use cases across a catalog of hundreds of millions of tracks and episodes — from release-relation detection and automatic content placement to analyzing podcasts for policy violations.

Important bounds on these numbers. The '10x' and '3x' are Spotify's internal metrics with no external audit, and the post does not disclose the comparison base: it is unclear from what starting corpus and over what period the growth was measured. A third-party review in ZenML's LLMOps case database notes further gaps: the post gives no technical detail on prompts, few-shot approaches, or LLM fine-tuning, and the 'low cost' claim is not itemized — whether infrastructure, prompt-engineering time, and ongoing quality monitoring are included is unknown. We present the case with these caveats: the direction of the result is not in doubt; take the exact multipliers as self-reporting.

In our view, the case's main value is an architectural pattern that transfers to virtually any industry: LLM plus humans, linked by an agreement metric with automatic escalation. This is fundamentally more reliable than 'LLM instead of humans': the model provides scale, people provide calibration and adjudication of edge cases, and the agreement metric automatically decides who should handle each specific item. The feedback loop is double: contested cases not only raise label quality but also reveal where labeling guidelines are ambiguous.

Our second observation: Spotify treats labeling as a product with its own team, tooling, and metrics rather than a one-off procurement. In an era when generative model quality is limited not by architectures but by evaluation datasets, an annotation platform becomes infrastructure as critical as model serving — and Spotify's case was the first from 'big streaming' to show that publicly.

Lessons learned

1. GenAI product quality bottlenecks on data: an annotation platform is infrastructure as critical as model serving and deserves a dedicated team.
2. LLM + humans beats LLM instead of humans: the model labels in bulk, humans resolve contested cases — automatic escalation on agreement metrics links them into one pipeline.
3. Three tiers of expertise (annotator → quality analyst → project manager) scale volume and quality at the same time.
4. A central platform with reusable tooling outruns dozens of homegrown processes: new labeling projects are configured on top of ready infrastructure.
5. Tools must match the content type: audio and video need purpose-built annotation interfaces, not adapted text tools.
6. Real-time metrics (volume, throughput, agreement) turn labeling from a black box into a managed production process.
7. Read the multipliers critically: '10x' and '3x' are internal figures with no disclosed baseline; third-party reviews (ZenML) note missing detail on prompts and full cost accounting.

Frequently asked questions