Together AI agents speed up open LLMs by up to 2.6x in tokens per second within a day

Together AI, which serves trillions of tokens a day, has introduced Together Compiler (TKCC), a system in which AI agents continuously search for optimizations to run models (inference) on a specific model, GPUs and current traffic. The problem is that doing this by hand no longer scales: model architectures differ, hardware differs, and the load changes every day. Say you want to test a single tweak to an attention kernel.

Normally that means restarting an engine such as vLLM, SGLang or TRT-LLM, and a full startup takes 467 seconds, almost 8 minutes. TKCC starts the engine once and swaps code on the fly, so a test starts in 9 seconds. The result: in under 24 hours, the system squeezed out up to 2.

6x more tokens per second (TPS, the main inference speed metric) on the open models Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.

1-Flash. And tokens per GPU translate directly into the cost of running the service.

Why traffic keeps shifting

The team shares observations from production. Working hours in one country produce longer prompts than in another: 97 thousand tokens versus 70 thousand, a factor of 1.4. After a new model is released, people first chat with it and evaluate it, and about two weeks later prompt length grows by 80% as users move to agentic scenarios. There are also spikes: prompts suddenly double while the cache hit rate drops.

Each such shift changes which optimization wins. For MoE models, uneven expert load matters; for sparse attention, what matters is real input data. Fixed heuristics and classic ML compilers handle this poorly. Among the ten most popular open LLMs on OpenRouter, no two share the same architecture, and hardware ranges from H100 and B200 to B300, NVL72 and Groq.

How Together Compiler works

The system takes a traffic profile, a model and hardware as input. Several agents generate optimization variants for the inference engine and exchange ideas through a shared "message board."

The first idea is "persistent engines." The bottleneck of earlier agentic systems was experiment speed, and Together's table shows why: loading weights, warming up JIT kernels and autotuning eat up minutes per trial. A persistent engine starts once and hot-swaps the candidate into the model's forward pass. As a result, 252 times more experiments get through than with baseline agentic systems.

The second idea is a balance between measurement speed and accuracy. Testing individual kernels on random data is fast but misleading: for MoE and sparse attention, the winner depends on the real distribution of inputs. So the engines have two tiers. "Fast" ones load a few layers and suit frequent, low-noise measurements. "Slow" ones load many layers and check whether the gain holds on real data.

The third component is a custom cost model for choosing parallelism (TP, EP, DCP and other schemes for splitting a model across GPUs). It combines hardware spec-sheet characteristics, microbenchmarks and measurements of real forward passes, and quickly prunes weak configurations. According to the authors' charts, a choice based on specifications loses to a measured layout, and the best layout depends on the number of concurrent requests.

What the numbers show

• Up to 2.6x more tokens per second compared with the best published configurations in the SemiAnalysis InferenceX (AgentX) benchmark.
• Low-latency records at batch size 1: 1111 tokens/s on MiniMax M3.1-Flash and 1205 on DeepSeek v4.1-Flash. The parallelism configuration for these was chosen by the custom cost model, not by an InferenceX recommendation.
• On the canonical megakernel test, Llama 1B with batch size 1, TKCC is 1.7x faster than open megakernels and keeps its advantage as sequence length grows (on H100 and B200).
• Beyond LLMs: on open codebases for weather forecasting, differential equations, time series and biology, TKCC found larger gains than Claude Code with the /goal command on three of four models.

What this means for practitioners

For those who run their own inference, the signal is this: optimization is not a one-off config selection but continuous recompilation to follow traffic drift. More tokens on the same card means either a smaller GPU fleet or more headroom for load.

There are caveats. The figures come from Together itself on its own tasks, and there is no independent verification yet. For Trainium2, the team used a synthetic multi-turn dataset because the implementation does not yet support high concurrency with limited HBM memory. Whether TKCC will become an available product is unknown. But two principles can be adopted right now: don't restart the engine for every test, and validate optimizations on real data, not random data.