Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarX: @the_denv→ original

A Million Tokens on a Single H100 Drops From $18 to $0.80 at 32 Concurrent Requests

The same H100 can produce a million tokens (the word fragments a model assembles its answers from) for either $18 or $0.80, depending on how many requests run at once. In a measurement shared by @the_denv, Qwen3-32B delivers 68 tokens/s to a single user and 1560 tokens/s across 32 users combined, 23 times more in total. For an internal chatbot, that means the card sits underused whenever only one person is talking to it. Calculate the price per token at your real load, not just the hourly rental rate. The key numbers:

AI-processed from X: @the_denv; edited by Hamidun News
A Million Tokens on a Single H100 Drops From $18 to $0.80 at 32 Concurrent Requests
Source: X: @the_denv. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @the_denv ↗
Editor-in-Chief
1 user68 tokens/s, about $18 per 1M output tokens
32 users1560 tokens/s combined, about $0.80 per 1M
Key metric #3Each of the 32 users gets about 49 tokens/s, versus 68 when alone

Imagine you are deploying an internal chatbot on the open Qwen3-32B model and running it on a single NVIDIA H100 GPU. A token is a fragment of a word; the model builds its answer from these fragments, and cost is conventionally counted per million of them. According to a measurement shared by @the_denv, when one person queries the bot, the card produces 68 tokens per second (tok/s), and a million output tokens cost roughly $18.

When 32 people write at the same time, the card delivers 1560 tokens per second in total, and a million tokens cost about $0.80. That is a 23x difference on identical hardware.

The practical takeaway: comparing GPU rentals by hourly price is not enough. You need to calculate the price per token at your actual load.

What exactly was measured

According to the post, with a single user an H100 running Qwen3-32B produces about 68 tokens per second. With 32 simultaneous users, total throughput rises to 1560 tokens per second. That roughly 23x gain comes without changing the hardware.

The cost of a million output tokens falls accordingly, from about $18 to about $0.80. The figures are consistent with each other. Working backward to the hourly price they imply gives about $4.4 per hour of card time. That is our own estimate; the post does not state a rate, but it looks like typical market rental pricing for an H100. The GPU costs the same either way, but the same money buys 23 times more tokens.

It is also worth working out what each user gets. 1560 tokens per second across 32 people comes to about 49 tokens per second each. A single request got 68. So per-user speed dropped by less than a third, while total output grew 23-fold.

Why this happens

The explanation is simple if you picture a bus. To produce each next token, the GPU has to re-read all of the model's weights from memory. For a 32-billion-parameter model, that is tens of gigabytes per step. With one user, the card spends almost all its time waiting for data to arrive from memory, while the compute units sit idle. It is like running an entire bus for a single passenger.

When many requests arrive, servers such as vLLM or SGLang merge them into a single batch. The weights are read from memory once, and all 32 users get their token from that one pass. The bus takes the same route, but now the seats are full. That is why total throughput grows almost linearly until the card hits a limit, either in compute or in the memory used for the context cache.

What this means in practice

First: the token price of a self-hosted model is not a constant. If you build your budget from a "single request in a quiet system" measurement, you may overestimate costs by tens of times. If you budget from peak load while real traffic is sparse, you will do the opposite and assume a price that is too optimistic.

Second: GPU utilization becomes the key metric. For internal services with a steady stream of requests, your own H100 can be cheaper than an API. For a product with occasional spikes, the card will spend most of its time in the "expensive" mode. In that case, a pay-per-token API or autoscaling with a queue often wins.

Third: you can batch more than just users. Background jobs such as summarization, classification, labeling, and running evaluations on test sets can be accumulated and sent in bulk. This is the cheapest way to get close to the numbers from the top-load mode.

Caveats

This is a single post with limited reach, not a published benchmark. The author does not specify which inference engine was used, what precision the model ran at (FP16, FP8, or more aggressive quantization), what the prompt and response lengths were, or what the time to first token was. All of these parameters shift the result significantly. Long contexts take up memory for the cache, so the number of users in a batch hits its ceiling sooner. Per-user speed will also depend on configuration.

So treat the figures as an order of magnitude, not a guarantee. But the direction is right and matches inference practice: token price is determined by hardware utilization. Before choosing between renting GPUs and using an API, run your own model on your own typical request profile at 1, 8, 16, and 32 parallel clients. Plot a curve of "speed per user versus cost per million tokens" and pick your operating point from it.

Story

Key Questions Answered

What exactly was measured?

According to the post, with a single user an H100 running Qwen3-32B produces about 68 tokens per second. With 32 simultaneous users, total throughput rises to 1560 tokens per second. That roughly 23x gain comes without changing the hardware. The cost of a million output tokens falls accordingly, from about $18 to about $0.80. The figures are consistent with each other. Working backward to the hourly price they imply gives about $4.4 per hour of card time.

Why this happens?

The explanation is simple if you picture a bus. To produce each next token, the GPU has to re-read all of the model's weights from memory. For a 32-billion-parameter model, that is tens of gigabytes per step. With one user, the card spends almost all its time waiting for data to arrive from memory, while the compute units sit idle. It is like running an entire bus for a single passenger. When many requests arrive, servers such as vLLM or SGLang merge them into a single batch.

What this means in practice?

First: the token price of a self-hosted model is not a constant. If you build your budget from a "single request in a quiet system" measurement, you may overestimate costs by tens of times. If you budget from peak load while real traffic is sparse, you will do the opposite and assume a price that is too optimistic. Second: GPU utilization becomes the key metric. For internal services with a steady stream of requests, your own H100 can be cheaper than an API.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…