Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarX: @raiharsh88→ original

Engineer sketches 10M tokens/s serving setup: 130 TB of cache and 700 nodes on MoE

For a chatbot to stream answers to 200,000 people at once at 50 tokens per second each, it has to produce 10 million tokens per second. The bottleneck is memory: the conversation cache for a 70B model at 4 thousand tokens of context comes to about 130 TB. By the estimate of engineer @raiharsh88, a DeepSeek-V3-class MoE model could handle this on 700 decode nodes, while a dense 70B needs 3–4 thousand H100s. Budget from memory, not from parameter count.

AI-processed from X: @raiharsh88; edited by Hamidun News
Engineer sketches 10M tokens/s serving setup: 130 TB of cache and 700 nodes on MoE
Source: X: @raiharsh88. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @raiharsh88 ↗
Editor-in-Chief
Key metric #1Prefill and decode run on separate GPU pools, with the cache shipped over RDMA
Key metric #2Speculative decoding and MTP heads give close to 2× per GPU
Key metric #3Routing requests to a replica that already holds the prefix skips most of the prefill

Picture a chatbot built on an open model, with 200,000 people writing to it at the same time and each receiving an answer at about 50 tokens per second (a token is roughly a word or a piece of one). In total, that is 10 million output tokens per second. Engineer @raiharsh88 published a breakdown of how he would build such a system, and his main conclusion is a surprise: the hardest part is not the model itself but memory.

Each of the 200,000 conversations has to store what has already been said — the so-called KV-cache — and together these caches take up about 130 terabytes. This is a back-of-the-envelope estimate, not a measured benchmark, but the arithmetic is useful for anyone planning their own inference.

First, the math: it all comes down to memory

The starting estimate is simple. 10 million tokens per second divided by 50 tokens per second per user gives about 200,000 concurrent streams.

Every stream has a KV-cache, the model's "working memory" of what has already happened in the conversation. For a 70B model with grouped-query attention (GQA) and an fp8 cache, that is roughly 160 KB per token. With a 4 thousand-token context, that works out to about 640 MB per conversation, and on the order of 130 TB across all 200,000. The author concludes that most of the design problem is memory, not compute.

How the author would build the system

Separating the stages. Prefill (the model reads your request) is compute-bound, while decode (it writes the answer one token at a time) is limited by memory bandwidth. Running both on the same GPUs hurts both stages. So the author proposes two separate GPU pools, with the finished cache shipped between them over RDMA, a fast network with direct memory access. DistServe and Mooncake take this approach.

Inside each GPU. Continuous batching (new requests join the batch on the fly), a paged KV-cache, fp8 weights, and fp4 on Blackwell. The batch size should be as large as the target latency allows. On top of that come speculative decoding or MTP heads (the model guesses several tokens ahead, then verifies the guess), and together they deliver close to a 2× speedup.

Choosing the architecture. The author bets on MoE: in such a model only a fraction of the parameters is active for each token, so each GPU yields more tokens. By his estimate, a DeepSeek-V3-sized model with wide expert parallelism would need about 700 decode nodes. A dense 70B model needs 3–4 thousand H100s for the same load.

Routing. In the author's view, it matters more than it looks. System prompts and long chats repeat heavily. Say every request shares the same long system prompt: if you send a new request to a replica that already has it in cache, most of the prefill is skipped. An idle cache can be offloaded to CPU RAM or NVMe, which is cheaper than recomputing it.

Operations and caveats

The author advises autoscaling on queue depth and KV-cache occupancy, not on CPU load. Loading a model takes minutes, so you need warm standby replicas. He suggests running in several regions and filling idle gaps in load with batch jobs that don't need a fast response.

What to keep in mind. All the figures are one engineer's estimates: the 700 nodes and 3–4 thousand H100s depend on response length, cache hit rate, target latency and hardware, and the author does not disclose all of his assumptions. Real systems face uneven traffic, spikes and failures, none of which are included in this estimate. A 4 thousand-token context is modest: for agentic scenarios with long conversations and tools, the cache will be noticeably larger.

Takeaways

For teams building their own inference, the practical conclusion is this. Budget from the memory needed for the KV-cache, not from the parameter count. Choosing an MoE model can cut the GPU fleet severalfold. Prefix caching and smart routing save money without buying new hardware. And separating prefill from decode is no longer exotic but a working setup that open projects already use.

For product people and founders, the lesson is simpler: the price of a token at large scale has an engineering basis. It is determined not by a model's impressive benchmarks but by how well the system manages memory, batches and cache.

Story

Key Questions Answered

What are the key details on First, the math: it all comes down to memory?

The starting estimate is simple. 10 million tokens per second divided by 50 tokens per second per user gives about 200,000 concurrent streams. Every stream has a KV-cache, the model's "working memory" of what has already happened in the conversation. For a 70B model with grouped-query attention (GQA) and an fp8 cache, that is roughly 160 KB per token. With a 4 thousand-token context, that works out to about 640 MB per conversation, and on the order of 130 TB across all 200,000.

How the author would build the system?

Separating the stages. Prefill (the model reads your request) is compute-bound, while decode (it writes the answer one token at a time) is limited by memory bandwidth. Running both on the same GPUs hurts both stages. So the author proposes two separate GPU pools, with the finished cache shipped between them over RDMA, a fast network with direct memory access. DistServe and Mooncake take this approach. Inside each GPU.

What are the key details on Operations and caveats?

The author advises autoscaling on queue depth and KV-cache occupancy, not on CPU load. Loading a model takes minutes, so you need warm standby replicas. He suggests running in several regions and filling idle gaps in load with batch jobs that don't need a fast response. What to keep in mind. All the figures are one engineer's estimates: the 700 nodes and 3–4 thousand H100s depend on response length, cache hit rate, target latency and hardware, and the author does not disclose all of his assumptions.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…