GPU Neural Network Optimization: How Together AI Speeds Up AI

Together AI applied GPU-level neural network optimization to show that the gap between theory and practice in AI is not inevitable but a direct challenge to engineers.

Why do neural networks run slowly on GPUs?

Because there is a gap between theory and practice: developers do not always account for every nuance of how a model interacts with the hardware. Together AI engineers closed that gap by adapting kernels for Blackwell GPUs in a week, work NVIDIA spent a full year on.

What is FlashAttention?

It is an optimization of transformer attention, published in May 2022. Its core idea: optimization of transformer attention was far from finished, and there were significant ways to speed it up.

What is FlashAttention and who developed it?

FlashAttention is a paper on optimizing transformer attention, published in May 2022 by Dan Fu and Tri Dao. The main idea: optimization of the attention mechanism was far from finished.

How long did

Together AI take to adapt kernels for Blackwell GPUs? One week. For comparison, NVIDIA spent a full year on it with dozens of specialists.

How can you make a neural network run faster?

Together AI uses kernel-level optimizations of transformer attention to adapt models to new GPUs. In a week, the team finished work that NVIDIA spent a full year on with dozens of specialists.