Together AI replaces megakernels with AI agents and speeds up Llama 1B by 28% on B200
Together AI has abandoned megakernels, single giant GPU kernels that run an entire model, and replaced them with Together Compiler, where AI agents rewrite and fuse kernels. On a B200, Llama 3.2 1B reaches 2118 tokens/s versus 1649 for the hand-written megakernel, a 28% gain. In production, the megakernel supported only one model, one batch size and one client. That makes automated search a better route to speed than manual assembly.
AI-processed from Together AI Blog; edited by Hamidun News
A year ago, Together AI was showing off megakernels: the entire forward pass of a model ran as one enormous GPU kernel, and the speeds were record-setting. Now the same team says it doesn't need them. In their place, the company has released Together Compiler, a system in which AI agents rewrite and fuse a model's kernels. On an NVIDIA B200, it squeezes out 2118 tokens per second on Llama 3.2 1B, while the hand-written megakernel manages 1649. That is a 28% difference. For anyone paying for inference, the value of this story is not the record but the breakdown of where speed actually comes from.
Why the megakernel never made it to production
The megakernel addressed three problems of ordinary inference. The first is the overhead of launching each kernel. In PyTorch, a single down-projection call on an H100 costs 37 microseconds of dispatch for 15 microseconds of actual GPU work. Even pure C++ adds about 2 microseconds per launch, and a single pass involves hundreds of launches. The second problem is the lack of overlap: the next layer's weights could be loaded while the current one is computing. The third is uneven utilization of streaming multiprocessors (SMs): some finish early and sit idle.
On Llama 1B, the megakernel delivered 990 tokens/s versus 669 for standard Torch Inductor, a 48% gain. In production, however, things looked worse. One researcher spent several months in 2025 integrating a paged KV cache, porting the code to Blackwell and chasing data races. The authors sum up the outcome this way: one model, one batch size (1), one precision (BF16) and one client. The speed is excellent, but there is no flexibility.
So the team worked backward: it took Torch Compile at 669 tokens/s and enabled the megakernel's optimizations one at a time. Adding CUDA Graphs brought launch cost down to roughly 1 microsecond and raised throughput to 864 tokens/s, or 87% of the megakernel. Adding PDL (an NVIDIA mechanism for overlapping adjacent kernels) on standard Inductor kernels got them to 98% of megakernel speed without writing a single custom kernel.
Where the standard compiler loses speed
The team studied Inductor on Llama 1B and found three weak spots.
First, its heuristics don't pick the best configuration for the workload. Inductor already has 607 tuning flags, 198 of them added in the past year. The freezing flag speeds up large batches by 10–26% but slows batch size 1 by 49%: the weights stay in 32-bit format, so the GPU reads twice as much data. Flags also interact with each other: aggressive_fusion alone cut speed by 6 tokens/s, but combined with three others it added 3.
Second, Inductor's benchmarks time kernels in isolation with a flushed L2 cache, while in a real pass the data is often already in cache. Isolated measurements came out 39–58% slower than live ones. Worse, in the embedding test the tokens are initialized with zeros, which makes the kernel look 2.1 times faster than it does on random data. As a result, Inductor chose a 7.4 µs Triton kernel for output-projection instead of a 6.4 µs cuBLAS kernel, which is 15% slower. On the MoE model GLM 5.3, random inputs in the benchmark inflated grouped-GEMM time by 4–21%.
Third, fusion is greedy and pairwise. Two adjacent kernels don't get fused because they don't share memory. But the first one launches only 6 blocks on a GPU with 132 SMs, leaving almost the entire chip idle. The fusion algorithm has to account for utilization, not just memory traffic.
What this means in practice
Three takeaways matter for production. First, before writing custom kernels, turn on CUDA Graphs and PDL: by Together's measurements, this recovers almost all of the megakernel's speed. Second, don't trust compiler flags blindly: the same flag behaves differently at different batch sizes, so you need systematic search rather than guesswork. Third, benchmarking should happen inside a real forward pass on real data, especially for MoE, where the input determines routing.
With agents and multi-kernels, the team beat its own old megakernels by up to 1.7x and reached more than 1100 tokens/s on Llama 1B on an H100. The idea behind Together Compiler is that agents continuously search for new fusion and configuration options tailored to a specific model, hardware and batch size, rather than relying on a single hand-written implementation.
There are caveats. All figures are the company's own measurements, taken on a very small 1B model, where launch overhead weighs especially heavily. On larger models the gain may be more modest, so you should verify the result on your own workload. Still, the direction is clear: hand-written megakernels remain a research tool, while in production, automated search across the space of optimizations wins.
- Nov 18, 2026OpenAI releases GPT-6.2 Astra: second place in the rankings and a 20% lower price
- Nov 18, 2026After a Failure, Agents Roll Back Their Work in Only 47% of Cases, Though They Solve 84% of Tasks
- Nov 12, 2026Meta* rolls out AI age checks on Instagram* worldwide: children under 13 will lose access
- Nov 12, 2026Meta brings Muse to Apple and Android watches with three ways to interact
Key Questions Answered
Why the megakernel never made it to production?
The megakernel addressed three problems of ordinary inference. The first is the overhead of launching each kernel. In PyTorch, a single down-projection call on an H100 costs 37 microseconds of dispatch for 15 microseconds of actual GPU work. Even pure C++ adds about 2 microseconds per launch, and a single pass involves hundreds of launches. The second problem is the lack of overlap: the next layer's weights could be loaded while the current one is computing.
Where the standard compiler loses speed?
The team studied Inductor on Llama 1B and found three weak spots. First, its heuristics don't pick the best configuration for the workload. Inductor already has 607 tuning flags, 198 of them added in the past year. The freezing flag speeds up large batches by 10–26% but slows batch size 1 by 49%: the weights stay in 32-bit format, so the GPU reads twice as much data. Flags also interact with each other: aggressive_fusion alone cut speed by 6 tokens/s, but combined with three others it added 3.
What this means in practice?
Three takeaways matter for production. First, before writing custom kernels, turn on CUDA Graphs and PDL: by Together's measurements, this recovers almost all of the megakernel's speed. Second, don't trust compiler flags blindly: the same flag behaves differently at different batch sizes, so you need systematic search rather than guesswork. Third, benchmarking should happen inside a real forward pass on real data, especially for MoE, where the input determines routing.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.