Pinterest: one embedding instead of three — +46.7% repins in Lens and a third of the ML systems to maintain

• Learning a Unified Embedding for Visual Search at Pinterest (KDD 2019) — Pinterest / arXiv (Zhai et al.), 2019-08-05
• Unifying visual embeddings for visual search at Pinterest — Pinterest Engineering Blog (Andrew Zhai), 2019-08-08
• Visual Discovery at Pinterest (история продуктов: Related Pins 2014, Similar Looks 2015, Flashlight 2016, Lens 2017) — Pinterest / arXiv (Zhai, Kislyuk et al.), 2017-02-15

Background

Pinterest is a platform for which visual search is not a feature but the essence of the product: users come to find ideas that often have no verbal description. The company built this capability over years: Related Pins (2014), Similar Looks (2015), Flashlight (2016) — find-similar inside any pin, Lens (2017) — search by a photo from the phone camera, and in 2019 the automated Shop the Look, which finds specific products inside a scene or outfit. By the time of the case study the platform had 250+ million monthly active users, a corpus of 200+ billion 'ideas', and visual search served more than 10 internal client products. The team itself put the query scale at 600+ million visual searches per month in the KDD 2019 paper (the accompanying blog post is more cautious: 'hundreds of millions').

The flip side of that evolution is the technical debt typical of fast-growing ML organizations. Each new product launched with its own specialized embedding model: the 2015 visual cropper ran on Caffe and VGG16, while 2018's Shop the Look used PyTorch and SE-ResNeXt. Stacks, datasets, and vector spaces diverged; it was easier for teams to spin up yet another task-specific model than to improve a shared one — and the fleet of specialized embeddings kept growing.

The visual search team led by Andrew Zhai described the fix in 'Learning a Unified Embedding for Visual Search at Pinterest', accepted at KDD 2019, and in an engineering blog write-up. It is one of the first detailed publications on consolidating several production embeddings into one universal embedding — with full offline and online experiment numbers.

Problem

Three visual search products — Flashlight, Lens, and Shop the Look — lived on three separate specialized embeddings, which meant tripled costs along the whole chain. Each model had to be trained and versioned separately; each needed its own retrieval and storage infrastructure; and crucially, every update to any model required re-indexing billions of catalog images into that model's vector space. Storage and serving costs grew linearly with the number of models while iteration speed fell: the team spent resources maintaining a zoo rather than improving quality.

There was a subtler problem too: improvements did not transfer. Progress in the Flashlight model did nothing for Lens and vice versa — knowledge learned on one product stayed locked in its embedding. Yet the tasks are genuinely different, which is what justified the original specialization. Flashlight finds similar items among 'clean' catalog-style images using engagement signals. Lens is a far harder problem: a camera photo — with household lighting, arbitrary angles, and noise — must match against the catalog domain; a classic domain shift. Shop the Look demands product-level precision: find not 'a similar chair' but this exact model.

The question the team set out to answer: can one universal embedding, trained on all three products' tasks at once, match the specialized models — or is quality degradation the unavoidable price of operational simplicity?

Solution

The team trained one universal embedding with multi-task metric learning. The base is an SE-ResNeXt101 convolutional network pre-trained on ImageNet; on top of the shared trunk sit per-task branches: a fully connected layer and a softmax classification loss. Training ran simultaneously on all three products' datasets: 800K Flashlight images (15K semantic classes assembled from engagement signals), 540K Lens images (2K human-labeled classes, mixed domains — from catalog imagery to camera photos), and 340K Shop the Look images (189 product classes plus 50K instance labels of specific items).

Two engineering modules made the scheme practical at Pinterest scale. A subsampling module enabled classification training over tens of thousands of classes by keeping the proxy bank on CPU and loading only the needed subset into GPU memory. A binarization module (with GroupNorm replacing LayerNorm, which proved critical for multi-task compatibility) compresses the vector into a binary code — it is the binary embeddings that ship to production, slashing storage costs and search latency across billions of images. The training infrastructure is PyTorch DistributedDataParallel with mixed precision (FP16, Apex): a typical run is 9 epochs on eight Tesla V100s, SGD with momentum 0.9.

The architecture's key product win: one vector space serves all search scenarios through shared retrieval infrastructure. One catalog re-indexing instead of three; one model improvement automatically reaches every product and the 10+ internal clients of visual search.

Validation ran on two levels. Offline — both automatic metrics on binary embeddings (for example, Shop the Look precision@1 rose from 33.0% with the specialized model to 52.8% with the unified one) and human relevance judgments on real queries. Online — an A/B test in Lens, the hardest domain: repins, clickthroughs, closeups, and engager propensity were measured.

Result

In offline human-judgment evaluations the unified embedding lifted precision@5 by +22.2% for Flashlight, +110.1% for Lens, and +72.1% for Shop-the-Look versus the specialized models. The hardest domain gained the most — Lens: multi-task training on mixed data provided exactly the generalization camera photos lacked. The paper also honestly shows the variance: within Shop the Look, per-category results range from −33.3% to +249.7% — an average win does not mean winning everywhere.

The online A/B test in Lens confirmed the offline picture with lifts in every product metric: repins +46.7%, clickthroughs +35.0%, closeups +32.7%, engager propensity up +16.3% to +26.7%. The operational bottom line: instead of three models, three training pipelines, and three retrieval infrastructures, the team maintains one; the paper states directly that deploying the unified embedding 'drastically reduced the operation and engineering cost' — while improving quality.

In our view, the case's key lesson is counterintuitive: model consolidation is usually seen as a 'simpler but worse' trade-off, yet at Pinterest the universal model beat the specialists at their own tasks. The mechanics follow ML theory — multi-tasking acts as regularization, and one product's data becomes augmentation for another — but the effect's scale (+110% in Lens) was possible because the tasks proved sufficiently related and the weakest domain received the most 'foreign' data. We would not generalize 'one embedding is always better' to any task set: the paper's own Shop the Look category spread demonstrates why.

Our second observation: this case is about the economics of ML platforms, not just quality. At 600+ million visual searches a month, every extra model means re-indexing billions of images and duplicate serving; a fleet one-third the size speeds up every subsequent iteration. Infrastructure consolidation is one of those rare investments that cut costs and raise team velocity at the same time; we note only that online numbers are published for Lens alone, and all results are company self-reporting.

Lessons learned

1. Model consolidation is both quality and economics: one multi-task embedding beat three specialized ones on relevance and cut the maintained infrastructure to a third.
2. Multi-task learning acts as regularization: one product's data improves the others — the hardest domain gained the most (Lens, +110% precision@5).
3. Look at variance, not just the mean: within Shop the Look, category results range from −33.3% to +249.7%; an average win does not guarantee winning every segment.
4. Validate both offline (human judges) and online (A/B): at Pinterest the precision gains were confirmed by lifts in every engagement metric.
5. Embedding binarization is an underrated economic lever: binary codes slash storage and search costs across billions of images; the devil is in the details (GroupNorm over LayerNorm).
6. Fewer models mean faster iterations: a single embedding improvement automatically upgrades every visual search product and 10+ internal clients at once.
7. Unification pays off at scale: with 600M+ visual searches a month, every re-indexing pass and every extra model is expensive.

Frequently asked questions