DuoLM Speeds Up LLM Recommendations 60x: 26,400 Scores per Second on a Single H100

LLM-based recommender systems run into an arithmetic problem. To score hundreds of candidates for a single user, the model has to process the full item descriptions together with the user's history, and server throughput runs out quickly. The authors of DuoLM — Jiang Wu, Yutong Song, Hongbo Wang, Guanghe Cao, and Jeff Lei Liang — propose eliminating the repeated work entirely. On a single H100 GPU, their system computes about 26,400 candidate scores per second. That is roughly 60 times faster than the teacher deployed on vLLM with automatic prefix caching, at comparable recommendation quality.

Two Models Instead of One Long Prompt

A typical LLM ranker takes everything as input at once: the item text, the user's history, and the user's reactions. For every user–candidate pair, the text has to be processed again, even though the description of a product or video does not depend on the user.

DuoLM splits the task into two parts. The first is an LLM-based encoder. It reads an item description and compresses it into a compact representation. This happens offline, and the result is cached and can be reused for any number of users. The second is a personalized ranker, also an LLM. It receives the ready-made candidate representations, the user's history, and feedback, and produces a recommendation.

Both models are trained jointly, end to end. There are two training signals: recommendation labels and distillation from an "expert" — a teacher that sees the full text. The encoder learns to compress descriptions so that the information loss does not hurt the final ranking, and the ranker learns to work with the compressed input.

Runtime: Virtual Tokens and CUDA Graphs

A model that saves compute in theory can easily lose the gain to overhead in practice. That is why the authors describe not only the architecture but also the serving side. It works like this:

• item representations are stored directly on the GPU and fed to the model through virtual token IDs — the ranker accesses the embeddings the same way it accesses ordinary tokens, without extra copying;
• several candidates within a single request share the computation of the user's history: its prefix is computed once, not once per candidate;
• CUDA Graph replay reduces kernel launch overhead, which matters when the computations themselves are short.

Together, the model and the runtime cut repeated work at every stage of the recommendation pipeline.

What 26,400 Scores per Second Gets You

Key parameters of the work:

• architecture: an LLM encoder for compact item representations and a personalized LLM ranker;
• training: end to end, recommendation labels and distillation from a full-text teacher;
• inference: virtual token IDs, shared history computation, CUDA Graph replay;
• hardware: a single H100 GPU;
• throughput: about 26,400 candidate scores per second;
• speedup: roughly 60x relative to the teacher on vLLM with automatic prefix caching;
• quality: comparable to the expert teacher model.

The comparison is interesting because the baseline is not a naive deployment but vLLM with prefix caching — in other words, an already optimized stack. Even that stack falls behind by an order of magnitude, because it caches prefixes but does not replace long item texts with short vectors.

For teams building LLM-based recommendations, the takeaway is practical. If your catalog changes more slowly than traffic grows (products, articles, videos, ads), it makes sense to convert descriptions into cacheable representations in advance. The online part then reduces to working with the user's history, and the cost of ranking a single candidate drops several-fold. This also eases the load on the GPU fleet: the same requests are served by fewer cards.

The limitations are also clear. The gain depends on the share of reused items and on how well the compressed representation preserves detail. The stated quality was measured on benchmarks against the teacher, not in a live A/B test on production traffic. Still, the overall scheme — an offline encoder, an online ranker, and a serving runtime built for them — looks reproducible: the project's code is published on GitHub, and you can check the 60x gap on your own data without rewriting your entire infrastructure.