Engineer sketches 10M tokens/s serving setup: 130 TB of cache and 700 nodes on MoE
For a chatbot to stream answers to 200,000 people at once at 50 tokens per second each, it has to produce 10 million tokens per second. The bottleneck is memory: the conversation cache for a 70B model at 4 thousand tokens of context comes to about 130 TB. By the estimate of engineer @raiharsh88, a DeepSeek-V3-class MoE model could handle this on 700 decode nodes, while a dense 70B needs 3–4 thousand H100s. Budget from memory, not from parameter count.
AI-processed from X: @raiharsh88; edited by Hamidun News
Picture a chatbot built on an open model, with 200,000 people writing to it at the same time and each receiving an answer at about 50 tokens per second (a token is roughly a word or a piece of one). In total, that is 10 million output tokens per second. Engineer @raiharsh88 published a breakdown of how he would build such a system, and his main conclusion is a surprise: the hardest part is not the model itself but memory.
Each of the 200,000 conversations has to store what has already been said — the so-called KV-cache — and together these caches take up about 130 terabytes. This is a back-of-the-envelope estimate, not a measured benchmark, but the arithmetic is useful for anyone planning their own inference.
First, the math: it all comes down to memory
The starting estimate is simple. 10 million tokens per second divided by 50 tokens per second per user gives about 200,000 concurrent streams.
Every stream has a KV-cache, the model's "working memory" of what has already happened in the conversation. For a 70B model with grouped-query attention (GQA) and an fp8 cache, that is roughly 160 KB per token. With a 4 thousand-token context, that works out to about 640 MB per conversation, and on the order of 130 TB across all 200,000. The author concludes that most of the design problem is memory, not compute.
How the author would build the system
Separating the stages. Prefill (the model reads your request) is compute-bound, while decode (it writes the answer one token at a time) is limited by memory bandwidth. Running both on the same GPUs hurts both stages. So the author proposes two separate GPU pools, with the finished cache shipped between them over RDMA, a fast network with direct memory access. DistServe and Mooncake take this approach.
Inside each GPU. Continuous batching (new requests join the batch on the fly), a paged KV-cache, fp8 weights, and fp4 on Blackwell. The batch size should be as large as the target latency allows. On top of that come speculative decoding or MTP heads (the model guesses several tokens ahead, then verifies the guess), and together they deliver close to a 2× speedup.
Choosing the architecture. The author bets on MoE: in such a model only a fraction of the parameters is active for each token, so each GPU yields more tokens. By his estimate, a DeepSeek-V3-sized model with wide expert parallelism would need about 700 decode nodes. A dense 70B model needs 3–4 thousand H100s for the same load.
Routing. In the author's view, it matters more than it looks. System prompts and long chats repeat heavily. Say every request shares the same long system prompt: if you send a new request to a replica that already has it in cache, most of the prefill is skipped. An idle cache can be offloaded to CPU RAM or NVMe, which is cheaper than recomputing it.
Operations and caveats
The author advises autoscaling on queue depth and KV-cache occupancy, not on CPU load. Loading a model takes minutes, so you need warm standby replicas. He suggests running in several regions and filling idle gaps in load with batch jobs that don't need a fast response.
What to keep in mind. All the figures are one engineer's estimates: the 700 nodes and 3–4 thousand H100s depend on response length, cache hit rate, target latency and hardware, and the author does not disclose all of his assumptions. Real systems face uneven traffic, spikes and failures, none of which are included in this estimate. A 4 thousand-token context is modest: for agentic scenarios with long conversations and tools, the cache will be noticeably larger.
Takeaways
For teams building their own inference, the practical conclusion is this. Budget from the memory needed for the KV-cache, not from the parameter count. Choosing an MoE model can cut the GPU fleet severalfold. Prefix caching and smart routing save money without buying new hardware. And separating prefill from decode is no longer exotic but a working setup that open projects already use.
For product people and founders, the lesson is simpler: the price of a token at large scale has an engineering basis. It is determined not by a model's impressive benchmarks but by how well the system manages memory, batches and cache.
- Nov 20, 2026Ant Group's SingProbe: an 8.1M-Parameter Guard That Slows an LLM by Less Than 0.5%
- Nov 20, 2026Local GLM 5.3 Speeds Up AI on ESP32-S3 by 100%, Qwen 3.8 27B by 30%
- Nov 19, 2026MLPerf Training: 10,240-GPU clusters and 49% faster training in six months
- Nov 19, 2026CoreWeave comes in up to 65% cheaper than hyperscalers on a three-year Blackwell cluster
Key Questions Answered
What are the key details on First, the math: it all comes down to memory?
The starting estimate is simple. 10 million tokens per second divided by 50 tokens per second per user gives about 200,000 concurrent streams. Every stream has a KV-cache, the model's "working memory" of what has already happened in the conversation. For a 70B model with grouped-query attention (GQA) and an fp8 cache, that is roughly 160 KB per token. With a 4 thousand-token context, that works out to about 640 MB per conversation, and on the order of 130 TB across all 200,000.
How the author would build the system?
Separating the stages. Prefill (the model reads your request) is compute-bound, while decode (it writes the answer one token at a time) is limited by memory bandwidth. Running both on the same GPUs hurts both stages. So the author proposes two separate GPU pools, with the finished cache shipped between them over RDMA, a fast network with direct memory access. DistServe and Mooncake take this approach. Inside each GPU.
What are the key details on Operations and caveats?
The author advises autoscaling on queue depth and KV-cache occupancy, not on CPU load. Loading a model takes minutes, so you need warm standby replicas. He suggests running in several regions and filling idle gaps in load with batch jobs that don't need a fast response. What to keep in mind. All the figures are one engineer's estimates: the 700 nodes and 3–4 thousand H100s depend on response length, cache hit rate, target latency and hardware, and the author does not disclose all of his assumptions.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.