Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureAPI / Product · ActionableHacker News AI→ original

Cerebras Hands Prompt Reading to AMD GPUs and Gets 5x More Tokens per Megawatt

An AI model's reply is produced in two stages: first it reads the prompt (prefill), then it writes the answer one token at a time (decode). Cerebras has moved the reading to AMD Instinct GPUs and kept the writing on its CS-3 systems. On the GPT-OSS-120B endpoint, this produced 5x more tokens per megawatt than a CS-3-only setup, and generation speed did not drop. The provider can serve more customers from the same power budget.

AI-processed from Hacker News AI; edited by Hamidun News
Cerebras Hands Prompt Reading to AMD GPUs and Gets 5x More Tokens per Megawatt
Source: Hacker News AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHacker News AI ↗
Editor-in-Chief
Key metric #1Tokens per megawatt show how many responses a data center delivers per unit of power
Key metric #2Early access is already in place at AWS, Fireworks AI, Gimlet Labs and General Compute
Key metric #3The company wants to deploy the setup at hundreds of megawatts by the end of 2027

An AI model's reply is produced in two stages: first it reads your prompt (called prefill), then it writes the answer one token at a time (decode). Cerebras, a maker of giant AI chips, has split these stages across different hardware: AMD Instinct GPUs now handle prompt reading, while Cerebras CS-3 systems still handle writing the response. In production, this setup produced 5x more tokens per megawatt of electricity than a CS-3-only configuration, and generation speed did not suffer. For anyone paying for fast agent responses, this is about how much interactive AI a data center can deliver from limited power.

Why it makes sense to split the request in two

Imagine sending a large chunk of code to Copilot or Codex with a question. First, the model has to read all of that text at once and build an intermediate state, including the KV cache, which is the memory of what it has read. This is heavy computation, and ordinary GPUs handle it well.

Then the response writing begins. The model outputs tokens one by one, and each new token depends on the previous ones. At every step it re-reads its data from memory, so here memory speed matters more than raw compute.

Cerebras chips, the size of an entire silicon wafer, are strong on memory. According to the company, they have 350 times more on-chip fast SRAM and 5,400 times higher memory bandwidth than a typical GB300. That makes them well suited for decode. Reading long prompts on such a chip is less sensible, so it is logical to hand that part to other accelerators.

What production showed

Cerebras launched the setup in its own cloud. AMD Instinct processes the prompts, and CS-3 writes the responses. When reading moves to the GPUs, Cerebras chips are fully occupied with generation.

The measurement was taken on the shared GPT-OSS-120B endpoint on November 14, comparing throughput per megawatt before and after enabling the split. The result was a 5x increase relative to the CS-3-only setup. According to the company, token generation speed remained the same.

For the provider, this means more customers and more revenue from the same power budget. For users, it means the same fast responses with a lower risk of running into capacity shortages. The plans are ambitious: by the end of 2027, the company intends to deploy the setup at hundreds of megawatts in its own cloud and with partners. Alongside this, the company mentions the announcement of CS-4, expanded production, and planned data center capacity of 700 MW.

How the solution is offered to partners

Cerebras offers partners a reference architecture: the accelerators they choose for prefill and Cerebras systems sit in the same network. There are two ways to connect.

The first is the full software stack, which gets a service to market faster. It includes:

  • a framework that distributes requests between prefill and decode pools and monitors the capacity and health of replicas;
  • a prefill engine built on open-source vLLM or SGLang;
  • a decode engine from Cerebras;
  • KV Portal, which transfers the KV cache between the two parts over RDMA.

The second path is a decode-only SDK for those who prioritize control. A partner brings its own framework and its own prefill engine and connects them to the Cerebras decode engine and KV Portal. The kit includes a decode replica simulator and an example integration with an open-source framework. Early access has already been granted to Amazon Web Services, Fireworks AI, Gimlet Labs and General Compute.

What to keep in mind

All figures come from Cerebras's own material and are not an independent benchmark. The 5x gain was measured on a single endpoint, with the baseline being the CS-3-only setup. The text contains no comparison with pure-GPU solutions on cost per token. There is also no data on how much time the KV cache transfer between the two pools adds. The phrase "without loss of speed" refers to response generation.

The practical takeaway for developers is this: separating prompt reading from response writing is becoming standard inference practice, and vLLM and SGLang are already built into this scheme. When choosing a provider for interactive agents, compare not only price per token but also response speed under load. For those building their own data centers, this is a signal that hardware for prompt reading and for response generation can now be selected separately.

Story

Key Questions Answered

Why it makes sense to split the request in two?

Imagine sending a large chunk of code to Copilot or Codex with a question. First, the model has to read all of that text at once and build an intermediate state, including the KV cache, which is the memory of what it has read. This is heavy computation, and ordinary GPUs handle it well. Then the response writing begins. The model outputs tokens one by one, and each new token depends on the previous ones. At every step it re-reads its data from memory, so here memory speed matters more than raw compute.

What production showed?

Cerebras launched the setup in its own cloud. AMD Instinct processes the prompts, and CS-3 writes the responses. When reading moves to the GPUs, Cerebras chips are fully occupied with generation. The measurement was taken on the shared GPT-OSS-120B endpoint on November 14, comparing throughput per megawatt before and after enabling the split. The result was a 5x increase relative to the CS-3-only setup. According to the company, token generation speed remained the same.

How the solution is offered to partners?

Cerebras offers partners a reference architecture: the accelerators they choose for prefill and Cerebras systems sit in the same network. There are two ways to connect. The first is the full software stack, which gets a service to market faster.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Private AI Builders Community · Vibe Club

Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews

Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.

Join Vibe Club →Explore Hamidun Academy
What do you think?
Loading comments…