Together AI beats TensorRT-LLM by 31% in coding-agent benchmarks

Together AI has published the first production-oriented inference benchmarks for coding agents, and the results call into question most of the industry's usual tests.

Why standard benchmarks are useless

A classic inference benchmark measures a single user on a dedicated server. The numbers look impressive, but they say nothing about real operating conditions. In production, dozens or hundreds of requests compete at once for the same KV cache, memory bandwidth and GPU cycles. The more traffic, the more time to first token (TTFT) grows. At some point the system becomes useless, even before a formal failure. Different engines reach that point at very different load levels, and that is exactly what needs to be measured.

Together AI designed its test around this scenario: coding-agent workloads, long context, high concurrency and zero tolerance for latency degradation.

What makes coding agents a special workload

Coding-agent requests carry enormous context: the file being edited, surrounding code, conversation history and snippets from vector search. Input length ranged from 45 to 200 thousand tokens, simulating how a real development session grows. The average response was about 450 tokens: the agent writes a function, not a novel. This kind of load creates three problems that standard tests miss:

• TTFT sensitivity. The developer stares at a blank screen until the first token arrives. Trust in the tool is lost in that gap between sending the request and the start of streaming. Generation speed is secondary: once tokens are flowing, the experience feels fast.
• Concurrent long context. Dozens of developers sending requests of 80+ thousand tokens fill the KV cache at the same time. The scheduler loses room to maneuver, TTFT creeps up, and the system degrades long before a formal failure.
• Prefill-heavy profile. The load falls mainly on prefill rather than decode. Engines tuned for long generation do not get their usual advantage.

The test ran on 4× NVIDIA B200 for each engine.

Together Inference Engine results

Together Inference Engine was compared with TensorRT-LLM and other leading OSS engines on identical hardware. Under a production coding-agent workload, the results were as follows:

• +31% tokens per second (TPS) versus the closest OSS competitor
• 2x better TTFT at traffic saturation
• 76% lower cost than Anthropic's Claude Opus 4
• Stable latency under high concurrency, where competitors are already degrading

The gain came from full-stack optimization: ThunderMLA technology, rewritten custom CUDA kernels and end-to-end profiling on real traffic.

"Most benchmarks measure a single user on a dedicated server.

The numbers look magnificent. They are absolutely useless for reasoning about production," the Together AI blog says.

What this means

The gap between inference engines is huge under real load, and synthetic tests do not reveal it. For teams building AI assistants for developers, the choice of provider directly affects how many users get a fast response at the same time, and how many end up staring at a blank screen. Production-quality inference is no longer a technical detail but a competitive advantage.