Together AI Reports 90% Faster Training on NVIDIA Blackwell

Together AI has announced immediate access to GPU clusters accelerated by NVIDIA Blackwell, and introduced its own optimization stack tailored specifically to the new neural-network hardware architecture.

Results: 90% speedup over H100

In tests with a Llama model of 70 billion parameters, the Together AI team reached 15,264 tokens per second on a single GPU. That is almost twice the result on the previous-generation NVIDIA HGX H100, which processed 8,080 tokens per second in an optimized configuration.

The results come from an optimized version of TorchTitan combined with Together Kernel Collection, the company's own collection of optimized kernels. For context: the tests use BF16 precision (Brain Float 16, a trade-off between speed and accuracy that is now the standard for training large models). According to the company, speed will keep growing with additional optimizations that are still in development.

How it works: optimization at the architecture level

The speedup was made possible by deep optimization for a specific GPU architecture. Together AI developed a set of components that make full use of NVIDIA Blackwell's capabilities:

• Custom FP8 kernels running on 5th-generation NVIDIA Tensor Cores (high-performance compute units)
• Attention kernels that run 1.8 times faster than FlashAttention-3 (the current standard for an optimized attention mechanism)
• Integration with the open ThunderKittens library to make full use of the dedicated on-chip memory
• Distributed training algorithms adapted to the topology of the Quantum-2 InfiniBand network

Tri Dao, Chief Scientist at Together AI and creator of FlashAttention, said: "We optimize every layer of the AI stack to fully exploit the advances in GPU architecture. We especially like the new Tensor Cores and the microscaling format for accelerating inference. The combination of Together Kernel Collection with NVIDIA Blackwell redefines the standards for efficient training and inference at scale."

Test program and scaling

As part of an exclusive launch program, Together AI is inviting eight pioneering AI companies to get direct access to dedicated HGX B200 nodes and the chance to work with NVIDIA engineers and Together AI researchers. The goal is to accelerate workloads together and find further optimizations.

At the same time, the company is deploying tens of thousands of HGX B200 servers and full-rack GB200 NVL72 systems with NVIDIA Quantum-2 InfiniBand networking. This includes the previously announced cluster of 36,000+ GPUs for training next-generation models and agents.

What this means

For AI companies, the result is practical: training large models becomes cheaper and faster. With a 90% gain in speed, models that used to take weeks to train now take days. That substantially cuts capital spending on compute and shortens the experimentation cycle for new architectures.

For the market as a whole, it is a signal: the era of generic GPU services is fading. AI companies that write their own optimized kernels for a specific architecture (like Together AI with ThunderKittens) gain a competitive edge in speed and cost. That directly affects the price of training and, ultimately, the price of AI services for end users.