NVIDIA unveils DynoSim to tune LLM serving parameters

NVIDIA has unveiled DynoSim, a tool for automatically optimizing the configurations of large language model serving systems. It helps engineers find the best combination of dozens of parameters by simulating the Pareto frontier: the set of configurations in which improving one metric inevitably worsens another.

The problem: hundreds of variables

Tuning LLM serving is not a matter of one variable but of a whole system of interacting parameters. Every choice affects the others, and local optimization often just shifts the bottleneck to another part of the system. For example, adding more workers for parallel processing can hurt latency because of a memory shortage. Picking a different backend means reconfiguring the scheduler.

The main parameters that must be considered at the same time:

• Model backend choice (vLLM, TensorRT, TensorRT-LLM, others)
• Tensor parallelism shape (how computation is distributed across several GPUs)
• Balance between the prefill (context preparation) and decode (response generation) phases
• Number of worker processes and threads on the host
• Scheduler strategy (batch size, dynamic batching)
• Traffic routing policy between nodes
• KV cache behavior and memory management
• Autoscaling and horizontal scaling thresholds

Engineers used to find the best configuration by trial and error. That meant weeks of testing on expensive GPU hardware, high costs, and no way to check every combination.

The solution: Pareto frontier simulation

DynoSim automatically simulates the parameter space and builds a performance map. Instead of testing on real hardware, the tool uses a physical model of the hardware and software to predict latency, throughput and memory consumption.

The output of DynoSim is a Pareto frontier: a set of non-dominated configurations. For example, one setup might deliver 50ms latency at 1000 req/sec throughput, another 100ms at 2000 req/sec. The engineer chooses based on priorities: the first if low latency matters, the second if maximum throughput matters, and something in between if the goal is balance.

The process usually takes hours of computation rather than weeks of experiments on real hardware. This speeds up the development cycle and lets engineers check hundreds of parameter combinations.

What this means

Tools like DynoSim move LLM serving optimization from pure experimentation toward a scientific discipline. Companies can now choose configurations on solid grounds instead of half-blind trial and error. For large cloud services, even a small efficiency gain cuts costs by hundreds of millions of dollars a year, so tools like DynoSim are quickly becoming an industry standard.