NVIDIA Dynamo: Session-Based Infrastructure Management for Agentic Inference

Agentic workloads are reshaping serving architecture

Agentic workloads have fundamentally changed the shape of traffic hitting inference servers. A session opens with a massive prefill — often tens of thousands of tokens of system prompts, tool descriptions, and user-specific instructions. After the first call, the model receives appended context, which the session then resends in full with every subsequent turn.

The difference from a chatbot lies in how the work unfolds: a single task spawns many parallel model calls, parallel sub-agents with both short and long lifespans, and tools that run between turns. Most of a session's wall-clock time isn't spent on generation at all — it's spent on tool execution and outbound calls, while the context sits resident in the KV cache, occupying space in HBM without being used. Optimal serving depends on how much context the system keeps resident and reuses, and how many sessions it can serve concurrently without breaking latency SLAs.

Session ID as a unified primitive for program-aware infrastructure

Most open-source serving stacks are built for the standard chatbot case: they route, admit, and cache every request independently, with no understanding of the session tying it to previous turns. NVIDIA Dynamo unified management around a session-level identifier, transforming request-aware infrastructure into program-aware infrastructure.

Every request in an agent's trajectory carries:

• session_id — a unique, stable identifier for a single chain of reasoning and tool calls • parent_session_id — the identifier of the parent session (for tracking agent/sub-agent hierarchy)

Built-in support for coding agents (works with zero configuration):

• Claude Code passes `x-claude-code-session-id` (root) and `x-claude-code-agent-id` (children) along with `x-claude-code-parent-agent-id` • Codex uses `thread-id` and `x-codex-parent-thread-id` • OpenCode sends `x-opencode-session-id` and `x-opencode-parent-session-id`

Plugins for other harnesses (agent-plugins repository):

• Pi provider — for Dynamo's OpenAI-compatible endpoint • Hermes middleware — maps Hermes's `session_id` • OpenClaw provider — maps OpenClaw's `sessionId`

Custom harnesses simply pass `X-Dynamo-Session-ID` (and optionally `X-Dynamo-Parent-Session-ID`), and Dynamo normalizes them into `agent_context`, which the entire stack reads.

Tracing, replay, and backtesting

With the `DYN_REQUEST_TRACE=1` flag, Dynamo automatically logs every request without storing prompts and responses (those can be enabled separately). Once the response stream completes, it emits a `request_end` record containing:

• Session fields (session_id, parent_session_id) • Output token count and finish reason • KV cache metrics and tool names • A replay block with input length and sequence hashes (a format similar to Mooncake)

Since every record contains timestamps and finish reasons, you can see the distribution of tool calls and their connection to eviction behavior. Dynamo uses this data for backtesting and deployment optimization.

A captured trace becomes a reusable benchmark: it can be replayed as many times as needed without overloading the model, tools, or external APIs. The trace remains content-free thanks to sequence hashes that become synthetic block IDs, preserving prefix sharing across turns and sessions.

AISimulate runs the request graph against a simulated scheduler, router, and KV cache to compare workers, topologies, routing policies, and capacity without burning GPU resources. AIPerf replays the same graph against a real endpoint for final latency, throughput, and cache metrics.

Session-aware routing and throughput management

A standard request-level router (including the built-in KvRouter) makes placement decisions request by request: which worker has maximum cache overlap. That's the right unit for a single turn, but the wrong one for an agent under high concurrency, because it misses two factors:

• Cache-occupancy blowup: between turns, an agent's cache isn't freed — it sits in HBM, holding blocks while the agent runs a tool off the GPU. With N agents at step k, the aggregate working set is N × context_k, but a request-level router only sees a single turn. Once the working set exceeds HBM and DRAM, the engine evicts the cache of live sessions, and every subsequent turn pays for a full re-prefill instead of a cache hit.

• No backpressure: when a worker is overloaded, a request-level router can only cancel or queue in-flight requests — both options are worse than the alternative: pausing the session at a natural break point before its next turn, right when it's about to go idle.

Dynamo adapted the scheduler from the ThunderAgent paper (Kang et al., 2026) on top of the KV-aware router. Implemented in Rust, it owns the KvRouter directly and tracks actual prompt_tokens + completion_tokens from each response instead of estimating from raw bytes. As a native Dynamo plugin, it can be adapted to the shape of your own workload without modifying the underlying router.