NVIDIA TensorRT Now Scales Generative AI Inference Across Multiple GPUs
NVIDIA has updated TensorRT: the engine now supports inference across several GPUs at once. Its key optimizations, including kernel fusion, memory management and quantization, are preserved when the workload is distributed. The update responds directly to the growth of generative models, which long ago stopped fitting into the memory of a single accelerator.
AI-processed from NVIDIA Developer Blog; edited by Hamidun News
NVIDIA has updated TensorRT with native support for inference across multiple GPUs at once. Large generative models can now run in production without manual sharding and without losing the engine's key optimizations.
Why one GPU is no longer enough
Modern generative models are growing faster than video memory. Diffusion networks for video generation, multimodal LLMs with extended context and complex media-content pipelines have long outgrown 80 GB, the ceiling of the flagship H100. Inference engineers faced a harsh choice: split the compute graph by hand and lose TensorRT's optimizations, or move to third-party frameworks with lower throughput.
TensorRT is the de facto standard for production deployment on NVIDIA hardware. The engine optimizes compute graphs at the kernel level: it fuses operations, plans memory use and applies quantization, which gives the lowest latency and highest throughput among the available options. The problem is that all of these optimizations previously worked only within a single GPU.
What multi-device inference offers
The new capability lets TensorRT automatically distribute a model across several GPUs while keeping the full set of optimizations:
- Kernel fusion: fuses operations to minimize the overhead of transferring data between devices
- Memory planning: manages VRAM across GPUs intelligently, without redundant tensor copying
- INT8/FP8 quantization: applied to the entire compute graph, not just to parts of it
- Tensor parallelism: automatically splits model weights across devices, with no manual code changes
- Pipeline parallelism: different layers of the network run in parallel on different cards, increasing overall throughput
Previously, achieving a similar result required a complex combination of TensorRT with external tools, such as TensorRT-LLM or Triton Inference Server, and several weeks of engineering work. Now multi-device support is built into the engine itself.
Who needs this today
Teams building inference pipelines for media generation stand to gain the most: text-to-video systems, real-time content adaptation, interactive avatars and multimodal assistants. All of these tasks demand both a large model (meaning a lot of memory) and minimal latency (meaning no compromises on optimization).
The feature also changes the economics of inference in the cloud. Instead of wrestling with weight sharding across a GPU cluster and maintaining home-grown synchronization logic, teams can use the standard TensorRT API and get the same performance with lower development and maintenance costs.
The mid-size segment deserves a separate mention: companies that have two to four GPUs but no dedicated ML infrastructure team. For them, removing the barrier to multi-device inference is the biggest practical shift.
What this means
Scaling AI inference across multiple devices is moving from a "task for narrow specialists" to a "built-in engine feature." When TensorRT takes over distribution, the gap between a trained model and a scalable production service shrinks significantly. That directly affects which AI products mid-size teams can afford to launch.
- Nov 18, 2026ffmpeg-skill gives Claude Code and Cursor 42 local video-editing tools
- Nov 18, 2026Huang steps down as NVIDIA CEO, company loses more than $1 trillion in market cap
- Nov 18, 2026Google releases Gemini 4 Neon: 3rd place in AI ranking and tasks 32% cheaper
- Nov 18, 2026Devin Bot built an Android app on its own in 44 days: 1 134 PRs from 183 messages
Key Questions Answered
Why one GPU is no longer enough?
Modern generative models are growing faster than video memory. Diffusion networks for video generation, multimodal LLMs with extended context and complex media-content pipelines have long outgrown 80 GB, the ceiling of the flagship H100. Inference engineers faced a harsh choice: split the compute graph by hand and lose TensorRT's optimizations, or move to third-party frameworks with lower throughput. TensorRT is the de facto standard for production deployment on NVIDIA hardware.
What multi-device inference offers?
The new capability lets TensorRT automatically distribute a model across several GPUs while keeping the full set of optimizations: Kernel fusion: fuses operations to minimize the overhead of transferring data between devices Memory planning: manages VRAM across GPUs intelligently, without redundant tensor copying INT8/FP8 quantization: applied to the entire compute graph, not just to parts of it Tensor parallelism: automatically splits model weights across devices, with no manual code changes…
Who needs this today?
Teams building inference pipelines for media generation stand to gain the most: text-to-video systems, real-time content adaptation, interactive avatars and multimodal assistants. All of these tasks demand both a large model (meaning a lot of memory) and minimal latency (meaning no compromises on optimization). The feature also changes the economics of inference in the cloud.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.