vLLM Now Runs on Vera Rubin NVL72: 7.8x Higher Per-GPU Throughput

The vLLM project, the most popular open-source engine for large language model inference, has announced support for NVIDIA's Vera Rubin NVL72 rack-scale system. According to the project's blog, per-GPU throughput on the AgentX workload is 7.8x that of the previous generation, GB200 NVL72. The release includes day-0 model support, FlashInfer kernels tuned for the Rubin architecture, and locality-aware MoE routing.

The number is eye-catching, but first we need to understand what exactly was measured.

What is claimed

The comparison is made per GPU, not per rack. Both platforms are in the NVL72 class: 72 accelerators joined into a single NVLink domain. That makes the comparison formally fair: rack against rack, accelerator against accelerator.

The workload is AgentX. Judging by the name and context, it is an agentic inference scenario: long contexts, many sequential model calls, heavy use of caching. The original announcement gives no details, and that is the main gap. Model configuration, numeric precision (FP8, FP4 or mixed), batch sizes and latency constraints directly determine how far the figure carries over to your production environment.

Key parameters of the release:

• Platform: NVIDIA Vera Rubin NVL72 vs. GB200 NVL72.
• Engine: vLLM with day-0 Rubin support.
• Kernels: FlashInfer, tuned for Rubin.
• MoE: locality-aware expert placement.
• Metric: throughput per GPU.
• Result: 7.8x on the AgentX workload.

Where the speedup comes from

The authors name three contributing factors. The first is the Rubin hardware itself. The announcement does not separate how much of the gain comes from the hardware and how much from software, and that matters. If a significant part of the improvement comes from kernel and routing optimizations, some of the gain may eventually appear on Blackwell as well. The real gap between the generations would then be smaller than 7.8x.

The second is FlashInfer. This library implements attention and related operations, and the quality of such kernels determines how much of a chip's theoretical power turns into tokens. Kernels written for a specific architecture usually deliver the most noticeable gains on new hardware.

The third is locality-aware MoE. In mixture-of-experts models, tokens scatter across different GPUs, and inter-card communication often becomes the bottleneck. If experts are placed and routed so that most traffic stays among the nearest neighbors in the topology, the load on the interconnect drops. In agentic scenarios with long sessions, this optimization is especially noticeable.

Reading the fine print

The figure comes from vLLM's own blog, which means the benchmark was prepared by the engine's developers, not by an independent lab. There is no data on per-token latency or cost per million tokens, and in operation these are what determine the economics. Higher per-GPU throughput does not necessarily mean a proportional drop in price: a next-generation rack costs more, draws more power and requires more complex cooling.

Still, day-0 support matters in itself. New hardware used to wait months for mature software, and during that period real performance fell well short of the spec sheet. Here, the stack is ready before the racks arrive.

What teams should do

If you are planning to buy or rent Rubin, do not build 7.8x into your business plan. Ask the provider for a run on your model and your request profile, with the same latency constraints as in production. Compare cost per token, not tokens per second per card.

If you run on Blackwell, update vLLM and FlashInfer and re-measure your baseline: some of the software improvements may apply there too. For MoE models, check separately how experts are placed on your topology. For other teams, the takeaway is simpler: the inference stack is catching up with hardware faster than before, and waiting months for maturity after a chip release no longer seems necessary.