Modal speeds up Sakana Namazu generation by 35% with a draft model
Two things stand in the way of speeding up an LLM for a production API: expensive GPU memory and slow response generation. For Sakana Namazu, a model with roughly a trillion parameters, Modal combined selective weight compression with a small model that proposes continuations for the main model to verify. According to the team, latency per token fell by about 35%, and a single replica could handle roughly 40% more traffic before queues started growing.
AI-processed from Modal Blog; edited by Hamidun News
If your AI service is slow to respond, buying more GPUs may be an expensive workaround. Accelerators spend some of their time waiting, some work is repeated, and a small model intended to speed up generation may struggle to predict what comes next. In a Modal case study published on October 23, the team explains how it addressed these problems for Sakana Namazu, a model with roughly a trillion parameters designed for Japanese business tasks, web search, and code execution.
The headline result from the first phase: according to Modal, generation latency per token fell by about 35%. A replica could serve roughly 40% more traffic before queues started growing. Several separate changes underpin those figures, and for developers, the sequence of decisions is more useful than any promise of the same speedup.
Compress weights with real tasks in mind
Namazu uses a Mixture-of-Experts architecture: the model contains many specialized blocks, and a router selects which ones to activate. These blocks account for most of its parameters. Engineers converted the routed experts to NVFP4, a low-precision format for NVIDIA Blackwell GPUs. Attention, the router, and shared experts remained in BF16.
Why be so selective? Compression frees memory, but evaluation on familiar English benchmarks can miss a decline in Japanese responses. The compression settings were therefore tuned on conversations resembling Namazu's production tasks, with a large proportion in Japanese. According to the team, Japanese instruction-following performance remained within the tests' margin of error relative to the production version of the model.
Imagine an assistant that drafts business correspondence for Japanese clients. After compression, it might still perform well on English tasks but become less consistent in using the appropriate form of address. The practical lesson: your evaluation set should cover your product's language, style, and use cases. A single overall ranking is not enough.
The draft model needs to know your traffic
The second technique is speculative decoding. A small model proposes several subsequent tokens, and the large model verifies them in a single pass. The more proposals it accepts, the fewer generation steps it needs to run.
An off-the-shelf helper struggled to predict Namazu's Japanese business style: its token acceptance rate was below 40%. The team trained its own DFlash model on data representative of Namazu. One important detail: it preserved the original token IDs. Converting a response to text and then tokenizing it again can change token boundaries within tool calls and JSON, even when the visible string remains identical.
Once the API launched, it produced data for the next improvement. A new helper based on the DFlash2 architecture was trained on roughly a million real requests. The comparison used two identical serving configurations and a thousand held-out requests with user-specified generation settings. The average number of accepted tokens per step rose from 3.51 to 5.01, a 43% increase. p99 latency fell from 68 to 52 seconds. This metric captures slow responses: roughly 99% of requests finish within the specified time.
Queuing is part of response time too
The engineers also eliminated CPU–GPU waiting in the draft-verification loop, integrated faster compute kernels, and began reusing setup operations. Running on four GPUs required a consistent token-acceptance decision across devices: rare disagreements between them caused hangs.
They separately changed how requests were distributed across replicas. The router considers where computations for the beginning of a prompt are already cached while also tracking load. This reduced queues and spikes in time to first token, although the speed of subsequent generation itself did not change.
The case study has substantial limitations: the infrastructure provider published the results, the article includes no independent verification, and cost per request was not disclosed. Percentages from different optimizations cannot be added together, and a thousand requests cannot represent every future workload. For your own API, a sensible first step is to measure time to first token, generation speed, and queuing separately, then compare changes using identical requests. That will help establish whether you need a new helper, compression tuning, or a fix to load distribution.
- Jul 7, 2026Sakana AI launches Sakana Translate for Japanese, English and Chinese
- Nov 12, 2026NVIDIA Vera Doubles Modal Sandboxes Speed: Up to 4,281 Tasks per Minute on 150 Containers
- Jul 19, 2026Sakana AI launches Sakana Marlin, an agent that produces 100-page reports with slides
- Feb 7, 2026PaperBanana: Google teaches AI to draw charts that won't embarrass you in front of reviewers
Key Questions Answered
What are the key details on Compress weights with real tasks in mind?
Namazu uses a Mixture-of-Experts architecture: the model contains many specialized blocks, and a router selects which ones to activate. These blocks account for most of its parameters. Engineers converted the routed experts to NVFP4, a low-precision format for NVIDIA Blackwell GPUs. Attention, the router, and shared experts remained in BF16. Why be so selective? Compression frees memory, but evaluation on familiar English benchmarks can miss a decline in Japanese responses.
What are the key details on The draft model needs to know your traffic?
The second technique is speculative decoding. A small model proposes several subsequent tokens, and the large model verifies them in a single pass. The more proposals it accepts, the fewer generation steps it needs to run. An off-the-shelf helper struggled to predict Namazu's Japanese business style: its token acceptance rate was below 40%. The team trained its own DFlash model on data representative of Namazu. One important detail: it preserved the original token IDs.
What are the key details on Queuing is part of response time too?
The engineers also eliminated CPU–GPU waiting in the draft-verification loop, integrated faster compute kernels, and began reusing setup operations. Running on four GPUs required a consistent token-acceptance decision across devices: rare disagreements between them caused hangs. They separately changed how requests were distributed across replicas. The router considers where computations for the beginning of a prompt are already cached while also tracking load.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.