Airbnb: deep learning in search — +0.6% bookings from a new architecture and +14% bookings for new listings

• Applying Deep Learning to Airbnb Search (KDD 2019) — Airbnb / arXiv (Haldar et al.), 2018-10-22
• Improving Deep Learning for Airbnb Search (KDD 2020) — Airbnb / arXiv (Haldar et al.), 2020-02-13
• Listing Embeddings in Search Ranking (800 млн сессий, +21% CTR карусели) — Airbnb Engineering Blog (Mihajlo Grbovic), 2018-03-13

Background

Search ranking is Airbnb's core ML mechanism: it decides which of millions of listings a guest sees on the first results page — and therefore which hosts get bookings. Investment in search converts into money directly, which is why its evolution is documented by the team with rare candor for the industry: in two landmark KDD papers — 'Applying Deep Learning to Airbnb Search' (arXiv preprint October 2018, KDD 2019) and 'Improving Deep Learning for Airbnb Search' (arXiv February 2020, KDD 2020), both authored by Malay Haldar and colleagues.

These papers are atypical for the genre: instead of showcasing a 'breakthrough architecture', the team describes a years-long path of iterations with exact booking-gain percentages from online A/B tests — and dissects the failed experiments in equal detail. The authors state it plainly: 'deep learning was steep learning for us', and they distill their methodology into the principle that users lead and the model follows — first find the problem in real guest-behavior data, then change the architecture, not the other way around.

By the first paper, Airbnb already had mature ML infrastructure around search. Real-time personalization on listing embeddings had been in production since 2017: Mihajlo Grbovic's team trained 32-dimensional vector representations of homes on more than 800 million search click sessions across 4.5 million active listings — the Similar Listings carousel on these embeddings gained 21% CTR. Ranking itself was driven by a gradient-boosted decision tree (GBDT) model that had replaced a manual scoring function — a launch the team calls one of the largest step improvements in bookings in company history. The problems began after that success.

Problem

The first gradient-boosting model delivered a powerful jump, but gains then dried up: the team kept increasing effort while A/B tests came back neutral again and again. This is the classic plateau of a mature ML system — when feature and hyperparameter tuning stops moving the business metric, raising the question of a paradigm shift with an unpredictable outcome.

Airbnb's search problem also resists off-the-shelf recipes from the ranking literature. Users almost never repeat a query for the same 'item': a guest who visited Paris searches for Rome next. Supply is hard-constrained: one home for one set of dates can be booked by only one guest, so results cannot simply surface 'the most popular'. The marketplace is two-sided — ranking balances guests' and hosts' interests, and behavioral signals (clicks, saves, booking requests) are noisy and biased.

Two systemic biases deserved dedicated attacks. First, cold start: new listings with no interaction history lagged by about 6% NDCG versus established ones; the irony is that the very engagement features giving the model its power (removing them cost 4.5% NDCG) buried newcomers at the bottom of the results — and without new listings, supply growth suffocates. Second, position bias: guests book what is shown higher more often, so training data inflates the quality of top-position listings, and a model trained on such logs self-confirms old ranking mistakes.

Solution

The move to neural networks went in iterations, each validated by an online booking A/B test. The first attempt — a simple network with a single hidden layer of 32 ReLU units — proved booking-neutral against GBDT. The breakthrough came with a Lambdarank NN: a ranking loss on booked/not-booked pairs weighted by NDCG deltas. Then an ensemble where GBDT leaf indices and factorization machine predictions fed the network as features. The first paper's finale is a deep network: 195 features (after expanding categoricals into embeddings), two hidden layers of 127 and 83 ReLUs, trained on 1.7 billion impression-outcome pairs. The DNN delivered gains comparable to stacking all previous models — with a far simpler system.

The dead ends get equal space in the papers. Listing-ID embeddings — an industry-standard trick — led to overfitting: even the most popular home is booked at most 365 times a year, not enough signal per ID. Multi-task learning (predicting both bookings and long views) sharply increased views — and did not move bookings. Every dropout variant degraded offline metrics; residual connections, a Deep & Wide architecture, and attention networks 'failed to move the needle'. Separate lessons concern data and engineering: without feature normalization, training saturated mid-run, while converting the pipeline from CSV to Protobuf sped up training 17x and drove GPU utilization to ~90%.

The second paper (KDD 2020) is the next turn. A two-tower architecture split the model into a 'query tower' and a 'listing tower', each compressing its side into a 100-dimensional vector; relevance is the distance between vectors. That is not only quality (+0.7% NDCG online) but performance: scoring complexity drops from O(N·H·(Q+L)) to O(N·H_l·L + H_q·Q), cutting the 99th-percentile latency by 33%.

For cold start they built an engagement-estimation mechanism for new listings: in place of missing history, the model receives an interaction forecast computed from neighboring listings. Position bias was beaten with position dropout: the display position is fed to the model as a feature but randomly zeroed during training (the chosen rate — 0.15) so the model does not learn 'top of the page = good'. Failures are logged again: hard price monotonicity crashed bookings by 1.6%; the 'soft' version — beautiful offline — produced −0.67% in production, and the test was rolled back.

Result

The second wave's outcomes are fixed in online A/B tests. The two-tower architecture: +0.6% bookings and +0.75% revenue with −33% p99 scoring latency; a curious side effect — the average price of booked homes fell 2.3%, meaning the model got better at matching guests' price preferences. The cold-start mechanism lifted new-listing bookings by 14% (and their first-page impression share by 14%), adding +0.38% to overall bookings and making the supply side healthier. Position dropout brought another +0.7% bookings and an unexpected +1.8% revenue; bookings of boutique hotels — a segment hurt by the bias — rose 1.1%.

The scale must be read correctly: fractions of a percent here are not 'small results'. At Airbnb volumes, +0.6% bookings is a huge absolute number, and the sum of sequential gains (+0.6%, +0.38%, +0.7% from the second paper alone) compounds into a double-digit cumulative effect over the years. The team itself calls applying neural networks to search one of the company's biggest ML success stories. The bounds are also honest: all figures are the company's self-reported internal A/B tests, but the methodological detail and the published failures give these numbers more weight than a typical press release.

In our view, the pair of papers' main value is not the specific architectures (two-tower networks and position-bias mitigation long ago became industry standards) but the documented culture: the sole arbiter of every change was an online test on money (bookings), with offline metrics serving only as a hypothesis filter. The price 'soft monotonicity' case is telling: an intuitively right, offline-beautiful idea lost 0.67% of bookings in production — without online-test discipline it would have stayed in the system.

Our second observation: Airbnb effectively published a 'map of rakes' for everyone taking neural networks into search ranking — from ID overfitting to saturation without normalization. Companies retracing this path save months not on others' successes but on others' dead ends; in that sense, honestly publishing failures is a rare case of engineering altruism that doubles as employer branding.

Lessons learned

1. A plateau after early success is normal: when GBDT stopped delivering, the answer was a paradigm shift (neural networks), not endless feature tuning.
2. Validate every improvement with the online money metric (bookings), not just offline NDCG: price 'soft monotonicity' won offline and failed in production at −0.67%.
3. Cold start deserves its own mechanism: engagement estimation for new listings lifted their bookings by 14% and made the marketplace healthier.
4. Position bias corrupts training data: the simple position-dropout trick (rate 0.15) converted into +0.7% bookings and +1.8% revenue.
5. Architecture can pay for itself in performance: the two-tower network not only added bookings but cut p99 latency by 33% by reducing scoring complexity.
6. Standard tricks don't transfer blindly: listing-ID embeddings overfit (a home books at most 365 times a year), and dropout degraded offline metrics.
7. Publish failures too: Airbnb's papers are valuable precisely because they document dead ends (residual, Deep & Wide, attention, multi-task) that save other teams months.

Frequently asked questions