Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarX: @spheron→ original

TensorRT-LLM Beats vLLM on H100: 2780 vs 2400 Tokens/s on Llama 3.3 70B

Spheron compared three inference engines for running neural networks: TensorRT-LLM, SGLang and vLLM. Picture a chat service with 100 simultaneous users, served by a single H100 card running Llama 3.3 70B in the FP8 compressed format. TensorRT-LLM produced 2780 tokens (text chunks) per second against 2400 for vLLM, and 95% of users got their first word within 1.28 s versus 1.45 s. Choosing the engine yields extra speed without new hardware, but test it on your own prompts.

AI-processed from X: @spheron; edited by Hamidun News
TensorRT-LLM Beats vLLM on H100: 2780 vs 2400 Tokens/s on Llama 3.3 70B
Source: X: @spheron. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @spheron ↗
Editor-in-Chief
SGLang lands in the middle2460 tokens/s and 1.38 s to first token (p95)
Key metric #2vLLM takes 13% longer than TensorRT-LLM to deliver the first word
Key metric #3Engine versions and request lengths are not specified in the post; the numbers are a rough guide

When you run a large model in production, the cost of a single response depends not only on the graphics card but also on the software that drives that card. This software is called an inference engine: it decides how to batch requests, how to manage memory, and how quickly it delivers the first words of a response. The GPU cloud provider Spheron Network published a short benchmark of three popular engines on the same card. The gap between them turned out to be significant.

What Was Measured

The setup: one Nvidia H100 graphics card, the Llama 3.3 70B model in the FP8 compressed format (numbers are stored in 8 bits instead of 16, so the model takes up less memory and runs faster), and 100 simultaneous requests. Picture a chat service where a hundred people write at the same moment and one card answers them all.

Two things were measured. The first is throughput in tokens per second. A token is a chunk of text, roughly a syllable or a short word. The more tokens a server generates in total, the more users fit on one card and the cheaper each response becomes. The second is p95 time-to-first-token: the time within which the first word of the response appeared for 95% of requests. Put simply, it is how long almost any user waits before the model "starts typing." This number determines how responsive the service feels.

Results by Engine

TensorRT-LLM (Nvidia's engine) came first on both metrics: 2780 tokens/s and 1280 ms to first token.

SGLang took second place: 2460 tokens/s and 1380 ms. It trails the leader by roughly 12% in generation speed and by 100 ms in first-word latency.

vLLM, the most widespread open-source engine, showed 2400 tokens/s and 1450 ms. This is last place, but it trails SGLang only slightly: about 2.5% in throughput and 70 ms in latency.

The post says TensorRT-LLM beats vLLM "by about 13%." Recalculating, TensorRT-LLM leads on throughput by about 16%, while vLLM takes about 13% longer to deliver the first word. The 13% most likely refers to latency. The ranking is the same either way: TensorRT-LLM, then SGLang, then vLLM.

For scale: dividing 2780 tokens/s by one hundred requests gives about 28 tokens per second per user. That is enough for a chat, but it shows that a 70B model on a single card is operating close to its limit.

What to Do With This, and What Not to Trust

If you have steady high load and are willing to spend time on setup, TensorRT-LLM may deliver around 16% additional throughput without buying hardware. Across a large fleet of cards, that is meaningful savings. But the speed comes at a cost: Nvidia's engine usually requires a separate build and is tied to the vendor's ecosystem, while vLLM and SGLang are simpler to launch and update. This is a general observation, not a conclusion drawn from the post.

Now the limitations. This is a single post with three lines of numbers, not a full report. It gives no request or response lengths and no launch parameters, and the engine versions are not disclosed in the available text. That matters: vLLM and SGLang are updated frequently, and a 2–3% gap between them can easily shift from one release to the next. The measurements were published by a GPU rental provider, and they cannot yet be independently reproduced from this post.

Another caveat: only one configuration was tested, 100 concurrent requests on a single H100. At 10 requests, or across several cards, the picture may differ. A first-word latency of 1.3–1.4 seconds will not suit everyone, for example voice assistants.

In practice, treat these figures as a hypothesis and run your own requests through all three engines. Compare on real prompt lengths and real load, and pin the versions. If TensorRT-LLM's advantage holds, the switch will pay off. If it doesn't, it is simpler to stay on vLLM or SGLang, which are easier to maintain.

Story

Key Questions Answered

What Was Measured?

The setup: one Nvidia H100 graphics card, the Llama 3.3 70B model in the FP8 compressed format (numbers are stored in 8 bits instead of 16, so the model takes up less memory and runs faster), and 100 simultaneous requests. Picture a chat service where a hundred people write at the same moment and one card answers them all. Two things were measured. The first is throughput in tokens per second. A token is a chunk of text, roughly a syllable or a short word.

What are the key details on Results by Engine?

TensorRT-LLM (Nvidia's engine) came first on both metrics: 2780 tokens/s and 1280 ms to first token. SGLang took second place: 2460 tokens/s and 1380 ms. It trails the leader by roughly 12% in generation speed and by 100 ms in first-word latency. vLLM, the most widespread open-source engine, showed 2400 tokens/s and 1450 ms. This is last place, but it trails SGLang only slightly: about 2.5% in throughput and 70 ms in latency.

What to Do With This, and What Not to Trust?

If you have steady high load and are willing to spend time on setup, TensorRT-LLM may deliver around 16% additional throughput without buying hardware. Across a large fleet of cards, that is meaningful savings. But the speed comes at a cost: Nvidia's engine usually requires a separate build and is tied to the vendor's ecosystem, while vLLM and SGLang are simpler to launch and update. This is a general observation, not a conclusion drawn from the post. Now the limitations.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…