PyTorch Blog: NVIDIA Dynamo Learns to Serve AI Agent Sessions

Agentic applications change the load profile of an inference server. PyTorch Blog makes this point in a post on session-oriented inference on NVIDIA Dynamo. The announcement lists three characteristics: a large initial prefill, repeated calls to the model within a single session, and subagents that run in parallel. Engineers know each of these individually, but together they break the familiar "request arrives, request is served" model.

The available portion of the post contains no benchmarks, so we give no specific speedup figures here. Below is a breakdown of what is known about the problem and the platform.

Why an agent is not like a chat

A single chat turn is a prompt, a response and a pause while the user reads. An agent works differently. It receives a system instruction, tool descriptions, history and, possibly, large chunks of code or documents. The model processes all of this in the first step, and that is a heavy prefill. The agent then calls a tool, gets the result, appends it to the context and calls the model again. The prefix barely changes; only the tail grows.

If the server treats each call as independent, it recomputes the same prefix over and over, or routes the request to a worker that does not hold the right KV-cache. Parallel subagents make the problem worse: several branches share a common context but diverge in their continuations and compete for GPU memory.

What Dynamo does here

NVIDIA Dynamo is an open-source distributed inference framework that the company introduced as the successor to Triton Inference Server. It builds on ideas that matter especially for agents: splitting prefill and decode across separate GPU pools, routing based on KV-cache contents, and offloading the cache from GPU memory to cheaper storage tiers. It runs on top of engines such as vLLM, SGLang and TensorRT-LLM.

The post's idea is to add the concept of a session to these mechanisms. The scheduler knows that a series of calls belongs to one agent and can keep its context close to the compute, rather than guessing at matches on every request.

What changes for the server when moving from chat to agent:

• the unit of scheduling is the session, not the individual request;
• it pays to compute the large initial prefill once and reuse it;
• repeated calls with a growing context require cache-aware routing;
• parallel subagents create load spikes on the shared prefix;
• storing the cache between calls becomes a question of memory capacity and cost.

What to check on your own stack

If you run agents on your own models, start with measurements. Look at what share of tokens is the repeated prefix and how often the next call in a session lands on the same worker. If that share is high but cache hits are few, you are paying for redundant prefill.

The second step is prefix stability. It is best to keep the system prompt, the tool list and the message order unchanged: any edit at the start of the context invalidates the cache. The third step is to decide where to store the cache between calls and how much you are willing to pay to hold it while the agent waits for a tool response.

For teams that buy inference as a service, the takeaway is simpler. Ask your provider how it prices and stores cached context in long sessions. For agents, that line in the price list affects the final bill more than the price per million tokens.

Full implementation details and measurements are published in the original PyTorch Blog post. Before building conclusions into your architecture, check them against the figures in the primary source.