SGLang with Humming Kernels Cut Time to First Token by 7–23% and Output Throughput by 17–34%
MoE models like DeepSeek store their weights in 4 bits to fit into GPU memory, but the weights must be unpacked before each multiplication. The Humming kernels in SGLang do this directly in GPU registers. On eight H20-3e cards, the first token arrives 7–23% faster than with Marlin, and switching from BF16 to FP8 adds 17–34% in output throughput. Before choosing a backend, compare both on your own workload.
AI-processed from X: @lmsysorg; edited by Hamidun News
Large MoE (Mixture of Experts) models like DeepSeek are expensive to run: they take up a lot of GPU memory, while only a fraction of the experts is active for each request. To fit them into memory, the weights are stored in 4 bits. But before multiplication, these weights have to be converted back into a format the GPU understands, and if the kernels do this inefficiently, the entire gain disappears.
The LMSYS team, which develops the SGLang engine for running LLMs, has presented its solution: the Humming library. According to their measurements, on long prompts it reduces time to first token (that is, the latency before the response starts, the prefill phase) by 7–23% compared with Marlin, a popular backend for low-bit weights. And if FP8 instead of BF16 is chosen for the expert computations, output speed grows by 17–34%.
What Humming is and how it works
Humming is a set of low-bit kernels (small programs for the GPU) that are compiled on the fly (JIT) for a specific task. They work both for dense layers and for MoE layers on NVIDIA graphics cards from the Turing generation to Blackwell. It is not an optimization for one recent card, but a tool for different hardware fleets.
The most interesting path is described for Hopper. The expert weights remain packed in 4 bits. Only at the last moment, directly in the registers (the fastest memory on the chip), are the FP8 operands restored from them, right before the multiplication on the Tensor Core. An analogy: instead of unpacking an entire truckload of goods in the warehouse, the courier opens the box at the client's door. No memory is spent on the "unpacked" weights, and data is not shuttled back and forth unnecessarily.
Dense layers, meanwhile, stay on the native FP8 path: Humming does not touch them.
What the numbers show
The measurements were made on DeepSeek-V4.1-Flash-1017 on eight H20-3e GPUs with a TP8/EP1 configuration. The announcement contains two comparisons.
The first is Humming versus Marlin. Both backends used MXFP4 weights and BF16 expert activations. Humming accelerated prefill by a factor of 1.07–1.23, that is, by 7–23%, on prompts from 4K to 1016K tokens. The metric is time to first token. For services where users submit large documents or repositories, this is what determines the perceived speed.
The second is a comparison within Humming. On the same checkpoint, switching expert activations from BF16 to FP8 increased output throughput by 17–34% at 1, 32, and 64 concurrent requests. The gain is visible both with single requests and under load, so it is not an effect limited to large batches.
What this means in practice
If you run MoE models on your own GPUs:
- Marlin does not have to be the default choice. On long prompts, Humming won by 7–23% on time to first token.
- The activation format matters too. Switching experts from BF16 to FP8 yields higher output speed, which lowers the cost per token on the same machines.
- Support for cards from Turing to Blackwell simplifies life for teams with mixed fleets: the same kernels can be tried on different hardware.
Limitations
These are the authors' own results from a short announcement. The test was run on one model and one configuration: eight H20-3e with TP8/EP1. On other cards, other parallelization schemes, or other checkpoints, the numbers may differ. The announcement contains no data on how FP8 activations affect response quality, so it is worth running your own evaluation sets before production. The comparison with Marlin covers only prefill, not decoding.
Details were published by the Venus and SCT teams in a full post. A reasonable plan for engineers: take a typical workload, run it on SGLang with Marlin and with Humming, compare time to first token and tokens per second, and then separately verify quality with FP8 activations.
- Nov 21, 2026Skein Agent Erases Finished Stages From Context: 97% Success Rate, But More Expensive Than OpenCode
- Nov 21, 2026Open-Weight Inkling 2 Mini Matches Qwen3.8-27B on Independent Benchmark, Runs on 16 GB of Memory
- Nov 21, 2026The Programmer Stopped Being an AI Agent Dispatcher: Multica Takes Over the Handoff Routine
- Nov 21, 2026MLCommons Starts Measuring AI APIs Under Real Load: 7 Runs for Agents
Key Questions Answered
What Humming is and how it works?
Humming is a set of low-bit kernels (small programs for the GPU) that are compiled on the fly (JIT) for a specific task. They work both for dense layers and for MoE layers on NVIDIA graphics cards from the Turing generation to Blackwell. It is not an optimization for one recent card, but a tool for different hardware fleets. The most interesting path is described for Hopper. The expert weights remain packed in 4 bits.
What the numbers show?
The measurements were made on DeepSeek-V4.1-Flash-1017 on eight H20-3e GPUs with a TP8/EP1 configuration. The announcement contains two comparisons. The first is Humming versus Marlin. Both backends used MXFP4 weights and BF16 expert activations. Humming accelerated prefill by a factor of 1.07–1.23, that is, by 7–23%, on prompts from 4K to 1016K tokens. The metric is time to first token. For services where users submit large documents or repositories, this is what determines the perceived speed.
What this means in practice?
If you run MoE models on your own GPUs: Marlin does not have to be the default choice. On long prompts, Humming won by 7–23% on time to first token. The activation format matters too. Switching experts from BF16 to FP8 yields higher output speed, which lowers the cost per token on the same machines. Support for cards from Turing to Blackwell simplifies life for teams with mixed fleets: the same kernels can be tried on different hardware.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.