Together AI: GPT-5.5, Gemini and Opus Can't Write Fast Multi-GPU Kernels

Together AI has published ParallelKernelBench (PKB), the first open benchmark that measures how well language models can generate efficient CUDA kernels for multi-GPU environments. After testing more than 40 models on 87 real-world tasks, the researchers found that the best frontier models solve fewer than a third of the tasks correctly, and only a handful of those solutions actually outperform a naive PyTorch implementation.

Why multi-GPU is harder

Language models have become reasonably good at writing code for a single GPU, and most existing GPU-programming benchmarks stop at that scenario. But real production AI systems moved past that limit long ago: they run on dozens or hundreds of GPUs at once. In such configurations, compute stops being the main bottleneck; communication between devices takes its place.

According to Together AI, the overhead of moving data between GPUs eats up more than 20% of latency during inference, and the gap will keep widening because chip performance continues to outpace the bandwidth of inter-chip interconnects.

Multi-GPU generation differs fundamentally from single-GPU generation for three reasons:

• Combinatorial explosion of options: the model must choose among tensor, context, expert, data and other forms of parallelism, each of which creates its own communication pattern
• A different performance model: instead of the local compute roofline, inter-chip interconnect bandwidth becomes the main limit
• A new architectural choice: how to physically move data between GPUs (via copy engine, TMA, SM load/store or NVLS) and whether to overlap data transfer with computation

How the benchmark works

PKB includes 87 tasks from real codebases (Megatron-LM, DeepSpeed, DeepEP, TensorRT-LLM and NeMo-RL), as well as non-standard distributed workloads: routing in graph neural networks, distributed FFT and Gaussian splatting. This selection covers all the main sharding approaches: tensor, context, data, expert, sequence and FSDP/ZeRO.

Each task starts with a standard PyTorch + NCCL implementation and a description of the hardware topology. The model must replace it with its own CUDA kernel that transfers data directly between GPUs over NVLink through symmetric memory, bypassing the standard collective-operations stack.

Scoring uses three criteria: correctness of the result, wall-clock speedup, and reaching the communication roofline, the theoretical limit of interconnect bandwidth.

Results and unexpected wins

Together AI tested more than 40 models, including GPT-5.5, Gemini 3 Pro and Opus 4.7, the current flagship models from OpenAI, Google and Anthropic. The results were equally disappointing across the board:

• The best model correctly solved fewer than a third of the 87 tasks
• Fewer than a quarter of the correct solutions beat the naive PyTorch + NCCL baseline
• The main failures came in managing communication between GPUs and choosing the right data-transfer method

Still, a few solutions were unexpectedly strong: some generated kernels outperformed every publicly available implementation. The most striking case is GRPO training in NVIDIA NeMo-RL. No optimized public code existed for this operation until now, and a language model wrote it before humans did.

"Several generated kernels turned out to be faster than anything available in the open," from the

ParallelKernelBench technical report.

What this means

PKB marks the next frontier in AI coding: the move from a single GPU to distributed multi-GPU systems. Frontier models can't handle it yet, but the rare flashes of success suggest progress is possible with targeted collection of specialized training data.

For teams optimizing inference and training on GPU clusters, this is an important reference point: the tooling is maturing, but it is not ready for broad use yet.