Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarTogether AI Blog→ original

Together AI replaces megakernels with AI agents and speeds up Llama 1B by 28% on B200

Together AI has abandoned megakernels, single giant GPU kernels that run an entire model, and replaced them with Together Compiler, where AI agents rewrite and fuse kernels. On a B200, Llama 3.2 1B reaches 2118 tokens/s versus 1649 for the hand-written megakernel, a 28% gain. In production, the megakernel supported only one model, one batch size and one client. That makes automated search a better route to speed than manual assembly.

AI-processed from Together AI Blog; edited by Hamidun News
Together AI replaces megakernels with AI agents and speeds up Llama 1B by 28% on B200
Source: Together AI Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTogether AI Blog ↗
Editor-in-Chief
Key metric #1CUDA Graphs and PDL on standard Inductor kernels deliver 98% of megakernel speed
Key metric #2On H100, agents and multi-kernels beat the old megakernels by up to 1.7x: 1100+ tokens/s
Key metric #3The freezing flag in Inductor sped up large batches by 10–26% but slowed batch size 1 by 49%

A year ago, Together AI was showing off megakernels: the entire forward pass of a model ran as one enormous GPU kernel, and the speeds were record-setting. Now the same team says it doesn't need them. In their place, the company has released Together Compiler, a system in which AI agents rewrite and fuse a model's kernels. On an NVIDIA B200, it squeezes out 2118 tokens per second on Llama 3.2 1B, while the hand-written megakernel manages 1649. That is a 28% difference. For anyone paying for inference, the value of this story is not the record but the breakdown of where speed actually comes from.

Why the megakernel never made it to production

The megakernel addressed three problems of ordinary inference. The first is the overhead of launching each kernel. In PyTorch, a single down-projection call on an H100 costs 37 microseconds of dispatch for 15 microseconds of actual GPU work. Even pure C++ adds about 2 microseconds per launch, and a single pass involves hundreds of launches. The second problem is the lack of overlap: the next layer's weights could be loaded while the current one is computing. The third is uneven utilization of streaming multiprocessors (SMs): some finish early and sit idle.

On Llama 1B, the megakernel delivered 990 tokens/s versus 669 for standard Torch Inductor, a 48% gain. In production, however, things looked worse. One researcher spent several months in 2025 integrating a paged KV cache, porting the code to Blackwell and chasing data races. The authors sum up the outcome this way: one model, one batch size (1), one precision (BF16) and one client. The speed is excellent, but there is no flexibility.

So the team worked backward: it took Torch Compile at 669 tokens/s and enabled the megakernel's optimizations one at a time. Adding CUDA Graphs brought launch cost down to roughly 1 microsecond and raised throughput to 864 tokens/s, or 87% of the megakernel. Adding PDL (an NVIDIA mechanism for overlapping adjacent kernels) on standard Inductor kernels got them to 98% of megakernel speed without writing a single custom kernel.

Where the standard compiler loses speed

The team studied Inductor on Llama 1B and found three weak spots.

First, its heuristics don't pick the best configuration for the workload. Inductor already has 607 tuning flags, 198 of them added in the past year. The freezing flag speeds up large batches by 10–26% but slows batch size 1 by 49%: the weights stay in 32-bit format, so the GPU reads twice as much data. Flags also interact with each other: aggressive_fusion alone cut speed by 6 tokens/s, but combined with three others it added 3.

Second, Inductor's benchmarks time kernels in isolation with a flushed L2 cache, while in a real pass the data is often already in cache. Isolated measurements came out 39–58% slower than live ones. Worse, in the embedding test the tokens are initialized with zeros, which makes the kernel look 2.1 times faster than it does on random data. As a result, Inductor chose a 7.4 µs Triton kernel for output-projection instead of a 6.4 µs cuBLAS kernel, which is 15% slower. On the MoE model GLM 5.3, random inputs in the benchmark inflated grouped-GEMM time by 4–21%.

Third, fusion is greedy and pairwise. Two adjacent kernels don't get fused because they don't share memory. But the first one launches only 6 blocks on a GPU with 132 SMs, leaving almost the entire chip idle. The fusion algorithm has to account for utilization, not just memory traffic.

What this means in practice

Three takeaways matter for production. First, before writing custom kernels, turn on CUDA Graphs and PDL: by Together's measurements, this recovers almost all of the megakernel's speed. Second, don't trust compiler flags blindly: the same flag behaves differently at different batch sizes, so you need systematic search rather than guesswork. Third, benchmarking should happen inside a real forward pass on real data, especially for MoE, where the input determines routing.

With agents and multi-kernels, the team beat its own old megakernels by up to 1.7x and reached more than 1100 tokens/s on Llama 1B on an H100. The idea behind Together Compiler is that agents continuously search for new fusion and configuration options tailored to a specific model, hardware and batch size, rather than relying on a single hand-written implementation.

There are caveats. All figures are the company's own measurements, taken on a very small 1B model, where launch overhead weighs especially heavily. On larger models the gain may be more modest, so you should verify the result on your own workload. Still, the direction is clear: hand-written megakernels remain a research tool, while in production, automated search across the space of optimizations wins.

Story

Key Questions Answered

Why the megakernel never made it to production?

The megakernel addressed three problems of ordinary inference. The first is the overhead of launching each kernel. In PyTorch, a single down-projection call on an H100 costs 37 microseconds of dispatch for 15 microseconds of actual GPU work. Even pure C++ adds about 2 microseconds per launch, and a single pass involves hundreds of launches. The second problem is the lack of overlap: the next layer's weights could be loaded while the current one is computing.

Where the standard compiler loses speed?

The team studied Inductor on Llama 1B and found three weak spots. First, its heuristics don't pick the best configuration for the workload. Inductor already has 607 tuning flags, 198 of them added in the past year. The freezing flag speeds up large batches by 10–26% but slows batch size 1 by 49%: the weights stay in 32-bit format, so the GPU reads twice as much data. Flags also interact with each other: aggressive_fusion alone cut speed by 6 tokens/s, but combined with three others it added 3.

What this means in practice?

Three takeaways matter for production. First, before writing custom kernels, turn on CUDA Graphs and PDL: by Together's measurements, this recovers almost all of the megakernel's speed. Second, don't trust compiler flags blindly: the same flag behaves differently at different batch sizes, so you need systematic search rather than guesswork. Third, benchmarking should happen inside a real forward pass on real data, especially for MoE, where the input determines routing.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…